Papers with language model
CWSeg: An Efficient and General Approach to Chinese Word Segmentation (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for Chinese word segmentation have achieved state-of-the-art performance, but they pose challenges in the deployment. |
| Approach: | They propose to augment PLM-based Chinese word segmentation schemes by developing cohort training and versatile decoding strategies. |
| Outcome: | The proposed model can be used to augment existing PLM-based models and improve their performance on Chinese LLaMA and Alpaca datasets. |
Continuous Learning in a Hierarchical Multiscale Neural Network (P18-2)
Copied to clipboard
| Challenge: | Language models are a major class of natural language processing (NLP) models whose development has led to major progress in many areas like translation, speech recognition or summarization. |
| Approach: | They propose a hierarchical multi-scale language model where short time-scale dependencies are encoded in the hidden state of a lower-level recurrent neural network while longer time- scale dependencies can be encoded into the dynamic of the lower- level network. |
| Outcome: | The proposed model uses a meta-learner to update the weights of the lower-level neural network in an online meta-learning fashion to prevent catastrophic forgetting in the continuous learning framework. |
K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained language models are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. |
| Approach: | They propose a knowledge-injected pre-trained language model that can be transferred to both natural language understanding and generation tasks. |
| Outcome: | The proposed model significantly outperforms baselines across the board in e-commerce scenarios. |
DynaMaR: Dynamic Prompt with Mask Token Representation (2022.emnlp-industry)
Copied to clipboard
Xiaodi Sun, Sunny Rajagopalan, Priyanka Nigam, Weiyi Lu, Yi Xu, Iman Keivanloo, Belinda Zeng, Trishul Chilimbi
| Challenge: | Recent research shows that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks. |
| Approach: | They propose an unsupervised approach to fine-tuning large language models using unsupervised approaches to many downstream tasks. |
| Outcome: | The proposed approach improves on four e-commerce applications and can achieve an average improvement of 10% in few-shot settings and 3.7% in data-rich settings over the standard approach. |
Identifying and Reducing Gender Bias in Word-Level Language Models (N19-3)
Copied to clipboard
| Challenge: | Existing discriminatory biases in training data can be amplified by models . text corpora exhibit socially problematic biase . |
| Approach: | They propose a metric to measure gender bias and a regularization loss term to minimize embeddings onto an embeddable subspace that encodes gender. |
| Outcome: | The proposed method reduces gender bias up to an optimal weight assigned to the loss term, and the model becomes unstable as the perplexity increases. |
Building Hierarchically Disentangled Language Models for Text Generation with Named Entities (2020.coling-main)
Copied to clipboard
| Challenge: | Named entities pose a unique challenge to traditional methods of language modeling. |
| Approach: | They propose a Hierarchically Disentangled Model for named entities in cooking recipes using a dataset from several publicly available online sources. |
| Outcome: | The proposed model is based on 158,473 cooking recipes from public sources. |
Leveraging Generative AI for Extracting Business Requirements from Legacy COBOL and PL/I Code (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing pipelines for extracting business requirements from legacy systems are difficult because they are scattered across interdependent programs and data definitions. |
| Approach: | They propose an LLM-augmented reverse-engineering pipeline that provides deterministic parsing and schema-constrainedLLM generation with bidirectional traceability. |
| Outcome: | The proposed pipeline achieves 93% agreement with expert-authored business rules and reduces documentation effort by approximately 70% on 3.4M lines across workloads. |
Improving Conversational Recommendation Systems’ Quality with Context-Aware Item Meta-Information (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to integrate the recommendation function and dialog generation function smoothly are lacking. |
| Approach: | They propose to integrate dialog context for recommendation and dialog generation better using a pre-trained language model and an item metadata encoder to integrate the recommendation and dialogue generation. |
| Outcome: | The proposed architecture improves the integration of recommendation and dialog generation functions. |
ABLE: Agency-BeLiefs Embedding to Address Stereotypical Bias through Awareness Instead of Obliviousness (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies in Natural Language Processing (NLP) have unveiled a concerning issue: stereotypical biases associated with demographic groups are prevalent. |
| Approach: | They propose an approach that actively encodes stereotypical biases into the embedding space by integrating stereotypes into a model that acquires agency and belief scores rather than directly representing stereotypes. |
| Outcome: | The proposed model can learn agency and belief stereotypes while preserving the language model’s proficiency. |
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations? (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are more proficient at code generation, but they still generate incorrect programs. |
| Approach: | They define a group of models that have a high log-probability and weak correctness checks. |
| Outcome: | The proposed model samples fail to understand counterfeits through three clear failure modes . counterfeits are confusing to the model as they are to other models, the authors say . |
A Partially Rule-Based Approach to AMR Generation (N19-3)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a representation of a sentence as a labeled graph . because of these abstractions, it can be difficult to generate from AMR back to a fluent English sentence . |
| Approach: | They propose a new approach to generating English text from Abstract Meaning Representation (AMR) it is largely rule-based, supplemented by a language model and simple statistical linearization models . they also address difficulties of automatically evaluating AMR generation systems . |
| Outcome: | The proposed approach produces a fluent English sentence with a high quality . it is difficult to generate from an AMR back to a sentence which preserves original meaning . |
Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models (e.g., InstructGPT) struggle to answer subquestions written for the incorrect options of the main questions, implying that they have a limited capability for explaining why incorrect alternatives should be eliminated. |
| Approach: | They crowdsource free-form rationale for main questions and generate subquestion texts that explain why we should select or eliminate answer options. |
| Outcome: | The proposed dataset shows that large language models struggle to answer the subquestions even if they are able to answer main questions correctly. |
HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs (2024.eacl-srw)
Copied to clipboard
| Challenge: | Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. |
| Approach: | They propose a framework that combines benchmarking LLMs’ hallucination tendencies with efficient hallucinian detection. |
| Outcome: | The proposed framework provides opportunities to test and improve LLMs and can generate benchmarking datasets tailored to specific domains. |
Grounded, or a Good Guesser? A Per-Question Balanced Dataset to Separate Blind from Grounded Models for Embodied Question Answering (2025.acl-short)
Copied to clipboard
Miles Shelton, Nate Wingerd, Kritim K Rijal, Ayush Garg, Adelina Gutic, Brett Barnes, Catherine Finegan-Dollak
| Challenge: | Embodied question answering (EQA) is based on using perception and action in an environment to answer natural language questions. |
| Approach: | They propose a "per-question balanced" EQA dataset that uses two different environments to ground a model's answers in its environment. |
| Outcome: | The proposed model performs better than chance on the PQB-EQA benchmark, showing that it does not require the model to use perception, let alone to act in its environment to find the answer. |
Fine-Grained Propaganda Detection with Fine-Tuned BERT (D19-50)
Copied to clipboard
| Challenge: | The goal of the Fragment Level Classification task is to detect and classify textual segments that correspond to one of the 18 given propaganda techniques in a news articles dataset. |
| Approach: | They propose a model that performs word-level classification using a pre-trained language model to detect and classify propaganda fragments in a news article dataset. |
| Outcome: | The proposed model performs word-level classification using a popular pre-trained language model. |
Coreference Resolution through a seq2seq Transition-Based System (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent coreference resolution systems use search algorithms to identify mentions and resolve coreference. |
| Approach: | They propose a text-to-text coreference resolution system that uses a semantic paradigm to predict mentions and links jointly. |
| Outcome: | The proposed system achieves state-of-the-art accuracy on CoNLL-2012 datasets with 83.3 F1-score for English, 68.5 F1 score for Arabic, and 74.3 F1 scores for Chinese. |
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
Biomedical relation extraction with pre-trained language representations and minimal task-specific architecture (D19-57)
Copied to clipboard
| Challenge: | Using a pre-trained BERT-Base model, we learn domain-specific language representations using biomedical text. |
| Approach: | They propose a system that extends BERT, a state-of-the-art language model, which learns contextual language representations from a large unlabelled corpus. |
| Outcome: | The proposed model outperforms a baseline model while relying on an extremely simple setup with no specially engineered features. |
Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing knowledge-grounded dialogue systems perform poorly on unseen topics due to limited topics covered in training data. |
| Approach: | They propose a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks. |
| Outcome: | The proposed language model generalizes well across knowledge-grounded dialogue tasks. |
Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2 (2021.acl-srw)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. |
| Approach: | They conduct experiments on an English essay dataset using Chinese-GPT2 . they find that the model can generate better continuations by learning to generate the in the fine-tuning stage. |
| Outcome: | The pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. |
CAVA: A Tool for Cultural Alignment Visualization & Analysis (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Using CAVA, researchers can analyze country-specific biases encoded in large language models. |
| Approach: | They propose a visualization tool that allows users to identify biases in language models by adding country-based questions and models. |
| Outcome: | The proposed tool can be used to analyze the cultural competencies of large language models across the dimension of geographic locales. |
Zuo Zhuan Ancient Chinese Dataset for Word Sense Disambiguation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a core task in natural language processing . ancient Chinese has rarely been used in WSD tasks due to lack of a dataset . |
| Approach: | They annotate ancient Chinese text Zuo Zhuan using a copyright-free dictionary . they apply a method to find the most appropriate sense in a context using k-NN . |
| Outcome: | The proposed dataset will be available on GitHub. |
NeuroPrompts: An Adaptive Framework to Optimize Prompts for Text-to-Image Generation (2024.eacl-demo)
Copied to clipboard
| Challenge: | Recent advances in text-to-image diffusion models have made it difficult to obtain high-quality images. |
| Approach: | They propose an adaptive framework that automatically enhances a user's prompt to improve the quality of generation models. |
| Outcome: | The proposed framework generates prompts similar to those produced by human prompt engineers and provides user control over stylistic features via constraint set specification. |
BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases (2023.acl-short)
Copied to clipboard
| Challenge: | Energy-based models (EBMs) have gained popularity for controlled text generation due to their high applicability to a wide range of constraints. |
| Approach: | They propose a language model with tunable biases to adjust the language model’s output logits. |
| Outcome: | The proposed model maintains the generator’s autoregressive nature to assert a strong control on token-wise conditional dependencies and overall fluency, and converges faster. |
DILBERT: Customized Pre-Training for Domain Adaptation with Category Shift, with an Application to Aspect Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pre-training can be sub-optimal in some cases . for example, aspect extraction tasks require domain and category invariant representations . |
| Approach: | They propose a domain-invariant learning scheme for BERT to fine-tune pre-trained language models on a source domain and then apply it to a different target domain. |
| Outcome: | The proposed scheme improves performance over state-of-the-art models while using fraction of the unlabeled data. |
How many words does it take to understand a low-resource language? (2025.naacl-srw)
Copied to clipboard
| Challenge: | We evaluated the documentation needed to create a sentence embedding space using widely spoken languages. |
| Approach: | They propose to use widely spoken languages as a proxy for low-resource languages to evaluate the documentation needed to create a sentence embedding space. |
| Outcome: | The proposed language model can be used to improve the performance of sentences embedded in low-resource languages. |
The Impact of Integration Step on Integrated Gradients (2024.eacl-srw)
Copied to clipboard
| Challenge: | Integrated gradients (IG) are a powerful tool for explaining the internal structure of a language model. |
| Approach: | They propose to customize the step count for each instance to minimize the error. |
| Outcome: | The optimal number of steps to maintain minimal error varies from instance to instance. |
SwahBERT: Language Model of Swahili (2022.naacl-main)
Copied to clipboard
| Challenge: | Social media and Internet forums are valuable sources of citizens’ opinions, which can be analyzed for community development and user behavior analysis. |
| Approach: | They present a pre-training and annotated datasets of Swahili and an emotion classification datasets that are manually annotating by two native Swahils. |
| Outcome: | The proposed model outperforms existing monolingual language model in almost all downstream tasks. |
Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing theoretical frameworks for large language models (LLMs) do not explain how pretraining leads to in-context learning. |
| Approach: | They propose a theoretical framework that allows LLMs to generalize to unseen instructions and perform in-context learning even when verbalizers are irrelevant to the task. |
| Outcome: | The proposed framework can be used to analyze LLMs' ability to perform in-context learning . it can be applied to linguistic, psychology, and philosophy tasks . |
A Controlled Reevaluation of Coreference Resolution Models (2024.lrec-main)
Copied to clipboard
| Challenge: | a pretrained language model is used in state-of-the-art coreference resolution models. |
| Approach: | They evaluate five coreference resolution models and control for language model used . they find that encoder-based CR models outperform decoder--based models in accuracy . |
| Outcome: | The encoder-based model outperforms the decoder--based models in accuracy and speed . older model generalizes the best to out-of-domain textual genres . |
CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction (2025.acl-industry)
Copied to clipboard
| Challenge: | Retrieval Augmented Generation (RAG) is a powerful application of Large Language Models (LLMs). |
| Approach: | They propose to use BERTScore to fine-tune Large Language Models on domain-specific data to improve citation accuracy. |
| Outcome: | The proposed approach improves citation accuracy by 15.46% with minimal latency and cost. |
A Simple Yet Effective Hybrid Pre-trained Language Model for Unsupervised Sentence Acceptability Prediction (2022.aacl-short)
Copied to clipboard
| Challenge: | Existing unsupervised prediction approaches rely on language models to estimate sentence acceptability . low-frequency words would have a significant negative impact on sentence likelihood . |
| Approach: | They propose a method that substitutes Part-of-Speech (POS) tags for low-frequency words in sentences . their method improves both a sentence acceptability benchmark and a cross-domain sentence evaluation corpus . |
| Outcome: | The proposed method improves on a sentence acceptability benchmark and a cross-domain sentence evaluation corpus. |
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts (2022.aacl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a longstanding NLP task that consists of identifying an entity in a sentence or document. |
| Approach: | They construct a dataset of seven entity types annotated over 11,382 tweets . they provide a set of language model baselines and analyze the performance of the model . |
| Outcome: | The proposed dataset contains seven entity types annotated over 11,382 tweets . the authors focus on short-term degradation of NER models over time and strategies to fine-tune a language model over different periods . |
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors (2024.eacl-short)
Copied to clipboard
| Challenge: | Using large language models to detect machine generated text is difficult for humans to distinguish between human-written and machine-generated text. |
| Approach: | They propose to use one language model to detect machine-generated text produced by another language model in a zero-shot way. |
| Outcome: | The proposed methods can detect machine-generated text without additional training/data. |
Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets (2024.findings-emnlp)
Copied to clipboard
Israel Azime, Atnafu Tonja, Tadesse Belay, Mitiku Yohannes Fuge, Aman Wassie, Eyasu Jada, Yonas Chanie, Walelign Sewunetie, Seid Yimam
| Challenge: | Low-resource languages are left behind due to the unavailability of resources. |
| Approach: | They propose to integrate task-specific and generative datasets to improve language model performance for Amharic by fine-tuning an Amharican instruction fine-to-tuned model. |
| Outcome: | The proposed model shows promising results in different NLP tasks and compares translated instruction datasets with the original model. |
More room for language: Investigating the effect of retrieval on language models (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented language models are a promising alternative to standard pretraining, but little attention has been put into understanding what this type of training scheme does to the underlying language model when analyzed as a standalone -separated from the overall retrieval pipeline. |
| Approach: | They propose an ‘ideal retrieval’ methodology to study these models in a fully controllable setting and propose a retrieval augmentation methodology to examine their effects. |
| Outcome: | The proposed model saves substantially less world knowledge in their weights, but is worse at comprehending global context. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
The ROOTS Search Tool: Data Transparency for LLMs (2023.acl-demo)
Copied to clipboard
Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, Anna Rogers
| Challenge: | a 1.6TB multilingual text corpus is currently the largest language model . large language models are ubiquitous in modern NLP, used directly to generate text and as building blocks in downstream applications. |
| Approach: | They propose a search engine for the 1.6TB multilingual ROOTS corpus offering both fuzzy and exact search capabilities. |
| Outcome: | The ROOTS Search Tool is an open-source search engine for the 1.6TB multilingual ROOTs corpus. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |
Relational Memory-Augmented Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing language models rely on word correlation and are difficult to interpret . existing models often lack explicit representations for such information . |
| Approach: | They propose a memory-augmented approach to condition autoregressive language models on knowledge graphs. |
| Outcome: | The proposed model improves perplexity and bits per character in an autoregressive language model . it is complementary to token-based memory and enables causal interventions . |
LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to language-based environment manipulation are difficult to generalize across environments. |
| Approach: | They propose a general framework for language-based environment manipulation tasks that can deal with various environments using the same generative language model. |
| Outcome: | The proposed framework achieves new state-of-the-art results on four of the tasks and the execution-guided pre-training strategy brings remarkable improvements on all experimental tasks. |
Aspect-Controlled Neural Argument Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Current argument generation models produce lengthy texts and allow the user little control over the aspect the argument should address. |
| Approach: | They propose a language model that can be controlled to generate sentence-level arguments for a given topic, stance, and aspect. |
| Outcome: | The proposed model generates high-quality arguments for argumentation and counter-arguments. |
Questions Are All You Need to Train a Dense Passage Retriever (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for dense retrieval require large supervised datasets with custom hard-negative mining and denoising of positive examples. |
| Approach: | They propose a new corpus-level autoencoding approach for training dense retrieval models that does not require labeled training data. |
| Outcome: | The proposed method matches or surpasses strong supervised performance levels on multiple QA benchmarks with no labeled training data or task-specific losses. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values (2023.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LMs) have been shown to be highly effective for identifying harmful training instances, but dataset size and model complexity constraints limit the ability to apply Shapley-based data valuation to fine-tuning large pre-trained language models. |
| Approach: | They propose an algorithm that aggregates Shapley values from subsets for valuation of entire training set and a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets and can filter fine-tuning data to increase language model performance compared to training with the full fine-uning dataset. |
Context-aware Stand-alone Neural Spelling Correction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing natural language processing systems are vulnerable to noisy inputs resulting from misspellings. |
| Approach: | They propose a stand-alone spelling correction problem that corrects the spelling of tokens without additional token insertion or deletion. |
| Outcome: | The proposed solution outperforms the state-of-the-art spelling correction model by 12.8% absolute F0.5 score. |
LLM-Rec: Personalized Recommendation via Prompting Large Language Models (2024.findings-naacl)
Copied to clipboard
Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Chris Leung, Jiajie Tang, Jiebo Luo
| Challenge: | Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. |
| Approach: | They propose a novel approach which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations. |
| Outcome: | The proposed approach improves recommendation quality and even basic MLP models achieve comparable or even better results than complex content-based methods. |
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained vision-language models do not possess the ability to conduct in-context learning. |
| Approach: | They propose to meta-train a language model to perform in-context learning on NLP tasks and then transfer this model to VL tasks by attaching a visual encoder. |
| Outcome: | The proposed model outperforms the baseline model on VQA, OK-VQA, and GQA while having 20 times fewer parameters. |
Is Encoder-Decoder Redundant for Neural Machine Translation? (2022.aacl-main)
Copied to clipboard
| Challenge: | Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. |
| Approach: | They propose to combine bilingual and multilingual translations to train a language model to do translation. |
| Outcome: | The proposed approach performs on par with the baseline encoder-decoder Transformer . the proposed approach is compared with the translation model in the target language . |
AMAN: Agent for Mentoring and Assisting Newbies in MMORPG (2025.coling-industry)
Copied to clipboard
| Challenge: | AMAN is a chatbot designed to help novice gamers learn the gameplay mechanics of online games. |
| Approach: | They propose a model that functions as a human-like chat buddy that interacts with users in a friendly manner while providing substantive informational depth. |
| Outcome: | The proposed model integrates continual pre-training with a sequence of online resources and instruction tuning on curated dialogues. |
LinkNav: Surfacing Interconnected Information in Scientific Articles (2026.acl-demo)
Copied to clipboard
| Challenge: | a non-linear reading order of academic literature is recognized by authors who make explicit connections between non-adjacent passages. |
| Approach: | They propose an enhanced reading experience which generates questions and searches for answer-bearing passages in academic papers to form intra-document connections when answers are found. |
| Outcome: | The proposed interface makes connections between related but non-adjacent passages even if the author did not make them explicit. |
Abstractive Document Summarization with Summary-length Prediction (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing abstractive summarization models do not consider summarizing-specific information such as the target summary length. |
| Approach: | They propose a method for enabling a model to understand summarization-specific information by predicting the summary length in the encoder and generating a summary of the predicted length in fine-tuning. |
| Outcome: | The proposed method improves ROUGE scores on the WikiHow, NYT, and CNN/DM datasets. |
Structured Discourse Representation for Factual Consistency Verification (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to verify factual consistency of text capture a performance gap compared with sentence-level entailment. |
| Approach: | They propose a method that combines structured discourse information extraction with a classifier, FDSpotter, for factual consistency verification. |
| Outcome: | The proposed method achieves competitive performance on two tasks: data-to-text generation and text summarisation. |
Do Multilingual Language Models Think Better in English? (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing studies show that translation-test improves performance of multilingual models by translating the input into English using an external machine translation system. |
| Approach: | They propose a new approach that leverages the few-shot translation capabilities of multilingual language models. |
| Outcome: | The proposed approach outperforms direct inference on 5 tasks. |
Extracting Latent Steering Vectors from Pretrained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. |
| Approach: | They propose to extract latent vectors directly from pretrained language model decoders without fine-tuning. |
| Outcome: | The proposed approach generates a target sentence nearly perfectly for English sentences . it outperforms pooled hidden states of models on a textual similarity benchmark . |
Representation Learning for Resource-Constrained Keyphrase Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art keyphrase generation methods depend on large annotated datasets, limiting their performance in domains with limited annotation data. |
| Approach: | They propose a method that first identifies salient information using retrieval-based corpus-level statistics and then learns a task-specific intermediate representation based on a pre-trained language model. |
| Outcome: | The proposed method improves keyphrase generation and zero-shot domain adaptation on multiple keyphrase benchmarks. |
Meta-learning via Language Model In-context Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have reduced "task learning and prediction" to a simple sequence prediction problem. |
| Approach: | They propose a meta-learning method that recasts task adaptation and prediction as a sequence prediction problem. |
| Outcome: | The proposed method outperforms MAML on two classification tasks and improves on binaryClfs. |
Cross-Modal Cloze Task: A New Task to Brain-to-Word Decoding (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work on decoding language from non-invasive brain activity is limited due to noisy nature of brain recordings. |
| Approach: | They propose a cross-modal cloze task to predict a word from a neural image . they use a pre-trained language model to leverage the pre-training language model . |
| Outcome: | The proposed method outperforms baselines on 20 participants from two brain imaging datasets. |
Improve Language Model and Brain Alignment via Associative Memory (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that associative memory is essential for language comprehension and comprehension. |
| Approach: | They propose to integrate associative memory into language models to improve alignment . they find alignment is improved in brain regions closely related to associativ memory processing . |
| Outcome: | The proposed model improves in brain regions closely related to associative memory processing. |
Bridging the Gap between Native Text and Translated Text through Adversarial Learning: A Case Study on Cross-Lingual Event Extraction (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent research in cross-lingual learning has found that combining large-scale pretrained multilingual language models with machine translation can yield good performance. |
| Approach: | They propose a model architecture that jointly encodes a source language input sentence with its translation to the target language during training and takes a target language sentence with it as input during evaluation. |
| Outcome: | The proposed model architecture can integrate machine translation to improve event extraction while adding machine-translated data yields unstable performance due to representational gap. |
Large Vocabulary Size Improves Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have investigated the properties of internal layers in large language models, but no studies have defined the vocabulary size. |
| Approach: | They propose a method to use a new vocabulary instead of the pre-defined one in a continual training scenario. |
| Outcome: | The proposed method outperforms the model with the pre-defined vocabulary in a continual training scenario. |
LangVAE and LangSpace: Building and Probing for Language Model VAEs (2025.emnlp-demos)
Copied to clipboard
| Challenge: | a novel framework for modular construction of variational autoencoders (VAEs) on pre-trained large language models (LLMs) is presented. |
| Approach: | They propose a modular framework for variational autoencoders on top of pre-trained large language models. |
| Outcome: | The proposed framework can encode pre-trained language models into more compact and semantically disentangled representations. |
Cue Me In: Content-Inducing Approaches to Interactive Story Generation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for automatic story generation focus on one-shot generation, but we focus on interactive story generation. |
| Approach: | They propose two ways to incorporate user-provided cue phrases into automatic story generation. |
| Outcome: | The proposed approach produces more topically coherent and personalized stories than baseline methods. |
ALERT: Adapt Language Models to Reasoning Tasks (2023.acl-long)
Copied to clipboard
Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissi, Siddharth Verma, Zhijing Jin, Gargi Ghosh, Mona Diab, Asli Celikyilmaz
| Challenge: | Large language models have shown increasing in-context learning capabilities with scaling up the model and data sizes. |
| Approach: | They propose a benchmark and suite of analyses to evaluate reasoning skills of large language models. |
| Outcome: | The proposed model compares pre-trained and fine-tuned models on tasks that require reasoning skills to solve. |
DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon (2022.tacl-1)
Copied to clipboard
Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, Emmanuel Dupoux
| Challenge: | Existing nonparametric models for text segmentation use a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Approach: | They propose a Bayesian nonparametric model that uses a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Outcome: | The proposed model improves on the Zero Resource Speech Benchmark 2017 and can learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark. |
Do Language Models Know When They’re Hallucinating References? (2024.findings-eacl)
Copied to clipboard
| Challenge: | State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. |
| Approach: | They propose to use hallucinated book and article references as "model organism" of hallucinism research . authors propose queries to the language model to identify hallucinous references . |
| Outcome: | The authors show that language models can identify hallucinated references without external resources . they show that LMs often produce inconsistent author lists for hallucinos, but also accurately recall the authors of real references . |
Probing Across Time: What Does RoBERTa Know and When? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to natural language processing rely on fixed artifacts such as language models . current studies have focused on how these models acquire and demonstrate knowledge . |
| Approach: | They apply probing techniques to examine how language models acquire knowledge . they aim to inform future work on more efficient pretraining and understanding dependencies . |
| Outcome: | The proposed model learns linguistic abstractions, factual and commonsense knowledge, and reasoning abilities fast, stably, and robustly across domains. |
Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddings (2023.emnlp-main)
Copied to clipboard
| Challenge: | XLMs can support cross-lingual transfer learning with little to no additional training data. |
| Approach: | They describe a mechanism for cross-lingual transfer learning by measuring the properties of the initial token embedding layer. |
| Outcome: | The proposed model can be used to support cross-lingual transfer learning . the initial token embedding layer is expressive and interpretable . |
You Truly Understand What I Need : Intellectual and Friendly Dialog Agents grounding Persona and Knowledge (2022.findings-emnlp)
Copied to clipboard
Jungwoo Lim, Myunghoon Kang, Yuna Hur, Seung Won Jeong, Jinsung Kim, Yoonna Jang, Dongyub Lee, Hyesung Ji, DongHoon Shin, Seungryong Kim, Heuiseok Lim
| Challenge: | Existing models that ground knowledge and persona at the same time are limited, leading to hallucination and a passive way of using personas. |
| Approach: | They propose a conversational agent that grounds external knowledge and persona simultaneously and a retrieval augmented generation model that generates utterances with lesser hallucination and more engagingness. |
| Outcome: | The proposed agent generates the utterance with lesser hallucination and more engagingness utilizing retrieval augmented generation with knowledge-persona enhanced query. |
A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for generating product descriptions from images are inaccurate and generic . e-commerce product descriptions are important for content marketing and increasing engagement . |
| Approach: | They propose a new setting for generating product descriptions from images, augmented by marketing keywords. |
| Outcome: | The proposed approach improves the accuracy and diversity of product descriptions by up to 3.3% on Rouge-L and 9.4% on D-5. |
Risks and NLP Design: A Case Study on Procedural Document QA (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models that answer recipes better than humans can mitigate risks to users. |
| Approach: | They propose to specialize the analysis to more concrete applications and their plausible users. |
| Outcome: | The proposed model answers recipes as well or better than humans who answered the questions on the web. |
In-Context Demonstration Selection with Cross Entropy Difference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. |
| Approach: | They propose a cross-entropy difference method for selecting in-context demonstrations that uses parameter efficient finetuning to train small models on training data. |
| Outcome: | The proposed method outperforms baseline selection methods on a mix-domain dataset and shows that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example. |
HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification (2024.eacl-long)
Copied to clipboard
| Challenge: | Hierarchical text classification is a complex subtask under multi-label text classification . the relevance of document sections can vary based on the hierarchy level, necessitating a dynamic document representation. |
| Approach: | They propose a text-generation-based framework that uses language models to encode dynamic text representations. |
| Outcome: | The proposed framework surpasses existing methods while handling data and mitigating class imbalance. |
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)
Copied to clipboard
| Challenge: | Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet. |
| Approach: | They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. |
| Outcome: | The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior. |
A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products (2022.coling-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models lack medicinal product knowledge for product vertical search. |
| Approach: | They propose a biomedical knowledge enhanced pre-trained language model for medicinal product vertical search using ELECTRA’s replaced token detection (RTD) pre-training. |
| Outcome: | The proposed model improves query-title relevance, query intent classification, and named entity recognition in query. |
LingGen: Scalable Multi-Attribute Linguistic Control via Power-Law Masking (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for controlling coarse attributes are less effective for finer-grained attributes and suffer from inefficiencies when many attributes must be handled jointly. |
| Approach: | They propose a controlled text generation model that allows fine-grained control over a large number of real-valued linguistic attributes. |
| Outcome: | The proposed model achieves the lowest average control error among evaluated methods while remaining efficient at inference and receiving the highest fluency scores in human evaluation. |
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that Transformer Encoder model can't automatically capture word order, so explicit position embeddings are required to be fed into the target model. |
| Approach: | They propose a Transformer-based language model DecBERT that uses a causal attention mask to capture word order. |
| Outcome: | The proposed model improves on the GLUE language understanding benchmark and accelerates the pre-training process. |
Analyzing the Understanding of Morphologically Complex Words in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Morphologically complex languages are challenging for NLP as a large amount of information is condensed into a single word, unlike in analytical languages where separate words make it easier to derive meaning. |
| Approach: | They use a Large Language Model to analyse compositional word formation and derivation to find ill-formed word forms. |
| Outcome: | The proposed model is capable of solving most tasks except identifying ill-formed word forms. |
Tracing Origins: Coreference-aware Machine Reading Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study has enriched pre-trained language models with syntactic, semantic and other linguistic information to improve their performance. |
| Approach: | They use a pre-trained language model to leverage coreference information to enhance word embeddings . they use additional encoder layers to focus on coreference mentions or a relational graph convolutional network to model the coreference relations. |
| Outcome: | The proposed model imitates the human reading process and leverages coreference information to enhance word embeddings. |
Zero-Shot Information Extraction as a Unified Text-to-Triple Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a number of information extraction tasks require task-specific training. |
| Approach: | They propose a text-to-triple translation framework for information extraction tasks . they propose enabling task-agnostic translation by leveraging latent knowledge of a pre-trained language model . |
| Outcome: | The proposed framework outperforms the existing methods on open information extraction tasks. |
COCO-DR: Combating Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using COCO-DR, we combat distribution shifts between source training tasks and target scenarios. |
| Approach: | They propose a method to combat distribution shifts between source training tasks and target scenarios by COtinuous COtrastive learning. |
| Outcome: | The proposed method outperforms existing models on BEIR and the giant GPT-3 embedding model with 500x more parameters. |
Grounded Compositional Outputs for Adaptive Language Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Language models are a key component of natural language processing, but their size is a problem because they are typically trained with a closed output vocabulary derived from the training data. |
| Approach: | They propose a fully compositional output embedding layer for language models that is grounded in semantically related words and free-text definitions. |
| Outcome: | The proposed model outperforms state-of-the-art methods and adaptation approaches on cross-domain modeling and cross-learning tasks. |
Language Model Pre-Training with Sparse Latent Typing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Modern large-scale Pre-trained Language Models focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences. |
| Approach: | They propose a new pre-training objective that enables the model to learn latent types . the objective allows the model a self-supervised way to extract sentence-level keywords . |
| Outcome: | The proposed model learns interpretable latent type categories without external knowledge and improves downstream tasks. |
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Chain-of-Thought prompting is a powerful technique for enhancing language model’s reasoning capabilities, but generating long and correct CoT trajectories is challenging. |
| Approach: | They propose to align the steps of Chain-of-Thought reasoning with loop iterations and apply intermediate supervision during the training of Looped Transformers. |
| Outcome: | The proposed method generates accurate reasoning chains for complex problems exceeding training length, and improves performance of the auto-regressive model. |
Generative Text Modeling through Short Run Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | Latent variable models for text capture global semantic and syntactic features when trained correctly. |
| Approach: | They propose a short run dynamics for inference that initializes from the prior distribution of the latent variable and runs a small number of Langevin dynamics steps guided by its posterior distribution. |
| Outcome: | The proposed model is able to generate coherent sentences with smooth transition and shows no sign of posterior collapse. |
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work has explored the suitability of pre-trained language models in low resource settings with less than 1,000 training data points. |
| Approach: | They propose to use pool-based active learning to speed up training while keeping the cost of labeling new data constant. |
| Outcome: | The proposed model can be fine-tuned to optimize for low-resource settings while keeping the cost of labeling constant. |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
DPTDR: Deep Prompt Tuning for Dense Passage Retrieval (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies show that prompt tuning is unfriendly for industrial deployment in dense retrieval tasks. |
| Approach: | They propose to apply prompt tuning to dense retrieval tasks to reduce deployment cost . they propose to use retrieval-oriented intermediate pretraining and unified negative mining . |
| Outcome: | The proposed method outperforms state-of-the-art models on MS-MARCO and Natural Questions. |
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing captioning models are mostly available on English or are specialised for other high-resource languages. |
| Approach: | They propose an image-blind few-shot multilingual captioning model that prompts a language model with retrieved captions. |
| Outcome: | The proposed model is competitive with fully-supervised multilingual captioning models without requiring any supervised training on captioning data. |
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning is effective in instruction learning for unseen tasks, but it relies on a large amount of human-annotated samples, which restricts its generalization. |
| Approach: | They propose an instruction tuning technique which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions and then tests its generalization ability on unseen tasks. |
| Outcome: | The proposed method improves IT performance versus labeled data and training tasks by constructing pseudo-labeled data from unlabele . data is used to build a model that can learn from human instructions for zero-shot generalization on unseen tasks. |
Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract relations from text use fine-tuned machine learning approaches. |
| Approach: | They introduce a framework that performs constrained cloze completion over pretrained language models to perform unsupervised relation extraction. |
| Outcome: | The proposed framework outperforms competing unsupervised relation extraction methods based on pretrained language models by 27.8 F1 points compared to the next-best method. |
Simple Hardware-Efficient PCFGs with Independent Left and Right Productions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scaling dense PCFGs to thousands of nonterminals have shown to be beneficial for unsupervised parsing, but they still perform poorly as a language model and as an unsupervised model. |
| Approach: | They propose a simple PCFG formalism with independent left and right productions that scales more effectively as a language model and as an unsupervised parser. |
| Outcome: | The proposed formalism scales better as a language model and as an unsupervised parser despite imposing a stronger independence assumption compared to low-rank approaches. |
Biomedical Event Extraction with Hierarchical Knowledge Graphs (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Biomedical event extraction is critical in understanding biomolecular interactions described in scientific corpus. |
| Approach: | They propose to integrate domain knowledge from Unified Medical Language System (UMLS) to a pre-trained language model using Graph Edge-conditioned Attention Networks and hierarchical graph representation. |
| Outcome: | The proposed approach achieves 1.41% F1 and 3.19% F1 improvements on the BioNLP 2011 GENIA Event Extraction task. |
Adapter Pruning using Tropical Characterization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on adapter pruning have not examined the optimal number of adapter parameters needed for downstream applications. |
| Approach: | They propose an adapter pruning approach that prunes adapter parameters without changing the orientation of underlying tropical hypersurfaces. |
| Outcome: | The proposed approach prunes adapter layers without changing the orientation of underlying tropical hypersurfaces. |
Stylometry in a Bilingual Setup (2020.lrec-1)
Copied to clipboard
| Challenge: | a stylometric method of comparing texts by most frequent words does not allow direct comparison of original texts and their translations, i.e. across languages. |
| Approach: | They propose a stylometric method that removes language-specific features and parses each language counterpart with a corresponding language model in UDPipe. |
| Outcome: | The proposed method removes language-specific features and keeps linguistically independent features of individual author signal. |
Exploring the zero-shot limit of FewRel (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to extract information from a language model are limited in their ability to generalize and do not perform as well as few-shot learning models. |
| Approach: | They propose a general purpose relation extractor that uses Wikidata descriptions to represent the relation’s surface form. |
| Outcome: | The proposed system is based on a FewRel 1.0 dataset, which provides an excellent framework for training and evaluating the proposed system in English. |
Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times. |
| Approach: | They propose to evaluate surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. |
| Outcome: | The proposed model variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, while smaller models show a ‘tipping point’ at convergence after the decrease in language model perplexity . |
N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking (2022.findings-acl)
Copied to clipboard
| Challenge: | augmentation of task-oriented dialogues has followed standard methods for plain-text despite its richly annotated structure. |
| Approach: | They propose an augmentation framework that utilizes belief state annotations to match turns from various dialogues and form new synthetic dialogues in a bottom-up manner. |
| Outcome: | The proposed framework performs better on seen values and more robust to unseen values on n-shot training scenarios. |
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle (2021.naacl-main)
Copied to clipboard
| Challenge: | Failing to capture the structure of input language could lead to generalization problems and over-parametrization. |
| Approach: | They propose a new syntax-aware language model that explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model. |
| Outcome: | The proposed model can achieve strong results in language modeling, parsing, and syntactic generalization tests while using fewer parameters than other models. |
Sentence Bottleneck Autoencoders from Transformer Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for pretraining a language model on text have been used for building models in NLP, but they do not work for sentence representations derived from pretrainer models based on tokens or basic pooling operations. |
| Approach: | They propose to build a sentence-level autoencoder from a pretrained transformer language model. |
| Outcome: | The proposed model achieves better quality than previous methods on text similarity and style transfer tasks while using fewer parameters than large pretrained models. |
MolXPT: Wrapping Molecules with Text for Generative Pre-training (2023.acl-short)
Copied to clipboard
| Challenge: | Experimental results show that Generative pre-trained Transformers (GPT) have great success in natural language processing. |
| Approach: | They propose a unified language model of text and molecules pre-trained on SMILES wrapped by text. |
| Outcome: | The proposed model outperforms strong baselines of molecular property prediction on MoleculeNet and performs comparably to the best model in text-molecule translation while using less than half of its parameters. |
Crawling The Internal Knowledge-Base of Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for representing factual knowledge in a language model are insufficient. |
| Approach: | They propose a procedure for “crawling” the internal knowledge-base of a language model by expanding a knowledge-graph around it. |
| Outcome: | The proposed method yields high precision graphs (82-92%) while emitting a reasonable number of facts per entity. |
Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on sentence prediction tasks uses shallow neural networks to learn essay representations and constrain calculated scores with regression loss or ranking loss. |
| Approach: | They propose to use a pre-trained language model to learn text representations first and then to constrain the scores with regression loss or ranking loss. |
| Outcome: | The proposed model outperforms state-of-the-art models on the Automated Student Assessment Prize dataset. |
Tracr-Injection: Distilling Algorithms into Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent efforts to characterize symbolic abilities of the transformer architecture have shown that the tasks that can be implemented in RASP are uncommon to learn from natural unsupervised data. |
| Approach: | They propose a programming language, called RASP, which can be directly compiled into transformer weights to implement these algorithms. |
| Outcome: | The proposed method improves out-of-distribution performance compared to baselines, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. |
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for directing language model outputs are limited in their accuracy due to a distributional gap . existing methods train static value functions on trajectories sampled exclusively from the base policy . |
| Approach: | They propose a framework to bridge a distributional gap in the accuracy of value functions . they propose RLHF to align language models with human values and task requirements . |
| Outcome: | The proposed framework reduces computational costs and improves value function accuracy by leveraging principled value function optimization. |
GLADIS: A General and Large Acronym Disambiguation Benchmark (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing acronym disambiguation benchmarks are limited to specific domains . a study on a Microsoft question answering forum found that only 7% of acronyms co-occur with their corresponding long forms, which confuses the readers about the meaning of a text. |
| Approach: | They propose a new acronym disambiguation benchmark with a dictionary and a pre-training corpus . they then pre-train a language model on the constructed corpus and show the challenges . |
| Outcome: | The proposed benchmarks pre-train a language model on the constructed corpus for general acronym disambiguation. |
Manipulating the Perceived Personality Traits of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Psychology research has long explored aspects of human personality like extroversion, agreeableness and emotional stability, three of the personality traits that make up the ‘Big Five’. |
| Approach: | They propose to use text generated from large language models to evaluate perceived personality traits and to frame them as tools for controlling personas in dialog systems. |
| Outcome: | The proposed models predict personality traits in different contexts and can be manipulated in a predictable way. |
Modeling Event Salience in Narratives via Barthes’ Cardinal Functions (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for estimating event salience without annotations are prohibitively costly because they require annotators to understand the concept of event salientity. |
| Approach: | They propose to use Barthes’ definition of event salience to compute event salientity without annotations by using a pre-trained language model. |
| Outcome: | The proposed methods outperform baseline methods on folktales with event salience annotation and fine-tuned language model is key factor in improving the methods. |
An Empirical Exploration of Local Ordering Pre-training for Structured Prediction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-training contextualized encoders with language model objectives is effective for structured prediction. |
| Approach: | They propose a semi-supervised method for pre-training contextualized encoders with language model objectives. |
| Outcome: | The proposed method is effective on three typical structured prediction tasks in four languages. |
You should evaluate your language model on marginal likelihood over tokenisations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural language models typically tokenise input text into sub-word units to achieve an open vocabulary. |
| Approach: | They propose that language models should be evaluated on their marginal likelihood over tokenisations instead. |
| Outcome: | The proposed approach is unsatisfactory and may bottleneck model out-of-domain performance. |
BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have impressive capabilities on a wide range of tasks, such as question answering and the generation of coherent text and code. |
| Approach: | They propose a framework for automatic evaluation of large language models on open-ended planning problems and a dataset of biology protocols with corresponding pseudocode representations. |
| Outcome: | The proposed framework evaluates an LLM on a dataset of biology protocols with corresponding pseudocode representations. |
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. |
| Approach: | They propose an unsupervised reference-free metric which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. |
| Outcome: | The proposed metric has higher correlations with human judgments while obtaining better generalization of evaluating generated texts from different models and with different qualities. |
UDAPTER - Efficient Domain Adaptation Using Adapters (2023.eacl-main)
Copied to clipboard
| Challenge: | Using adapters, unsupervised domain adaptation (UDA) is more parameter efficient and requires large-scale data to be effective. |
| Approach: | They propose to add small bottleneck layers to each layer of a pre-trained language model to make it more parameter efficient by adding adapters. |
| Outcome: | The proposed methods outperform unsupervised domain adaptation methods such as DANN and DSN in natural language inference and sentiment classification tasks. |
Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for creating vision-and-language models involve structural modifications and V&L pre-training. |
| Approach: | They propose to extend a language model through structural modifications and V&L pre-training to make it inherit the capability of natural language understanding from the original language model. |
| Outcome: | The proposed method improves performance of vision-and-language models by extending pre-trained models with the same pre-training. |
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages (2024.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models are weak in addressing cross-lingual transfer tasks. |
| Approach: | They propose a method for initializing embeddings and choosing the right vocabulary size for cross-lingual systems. |
| Outcome: | The proposed method improves the F1-Score in several languages . |
TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can produce informative and fluent text with the help of large-scale datasets, but they suffer insufficient learning problem with limited training data. |
| Approach: | They propose to use table transformation module with template to rewrite structured table in natural language as input for GPT-2 and exploit multi-task learning with two auxiliary tasks to preserve table’s structural information. |
| Outcome: | The proposed model outperforms existing systems on most few-shot settings. |
From ID to LLM: Rethinking Representation Learning for Recommendation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies indicate a fundamental incompatibility between ID representations and language model (LM) representations as they capture behavioral and semantic spaces respectively. |
| Approach: | They propose a Profile-then-Embedding framework for recommendation that integrates semantic user and item profiles and a Personalized Embedded stage to encode these profiles into task-aligned recommendation embeddings. |
| Outcome: | The proposed framework achieves significant gains across three benchmark datasets, including cold-start and long-tail scenarios. |
Context-Aware Language Modeling for Goal-Oriented Dialogue Systems (2022.findings-naacl)
Copied to clipboard
| Challenge: | Goal-oriented dialogue systems face a trade-off between fluent language generation and task-specific control. |
| Approach: | They propose a method to fine-tune language models in a goal-aware way . they evaluate a flight-booking method with a context-assisted language model . |
| Outcome: | The proposed method outperforms the state-of-the-art method on a flight-booking task by 7% in terms of task success. |
Exploring Language Model’s Code Generation Ability with Auxiliary Functions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Auxiliary function is a useful component to improve language model’s code generation ability, but a systematic exploration of how they affect has yet to be done. |
| Approach: | They construct a human-crafted evaluation set which contains examples of two functions where one function assists the other to examine their ability in a multifaceted way. |
| Outcome: | The proposed model is underutilized to call the auxiliary function, suggesting future directions to enhance their implementation by eliciting the supplementary function call ability encoded in the models. |
Syntax-guided Contrastive Learning for Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies rely on additional syntax-driven attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. |
| Approach: | They propose a syntax-guided contrastive learning method which does not change the transformer architecture and does not alter the transformer structure. |
| Outcome: | The proposed method achieves consistent improvements in a variety of tasks including grammatical error detection, entity tasks, structural probing and GLUE. |
DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. |
| Approach: | They propose a methodology to study the soundness of Social-IQ by applying simple perturbations to a dataset of multiple choice questions on videos of complex social interactions. |
| Outcome: | The proposed method reduces biases in the original dataset and improves performance. |
Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models lack visual knowledge of common objects due to reporting bias. |
| Approach: | They investigate whether integrating visual knowledge into a language model can fill the gap . they use captions and images to transfer visual knowledge to 5 downstream tasks . |
| Outcome: | The proposed model can improve performance on 5 tasks that may need visual knowledge to solve the problem. |
Eliciting Affective Events from Language Models by Multiple View Co-prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to generate training data using weakly labeled data are costly and limited . |
| Approach: | They propose a method for acquiring and labeling affective events with multiple view co-prompting using pre-trained language models. |
| Outcome: | The proposed approach improves state-of-the-art affective event classifier on two datasets. |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge representation learning methods do not use graph contextualized knowledge. |
| Approach: | They propose to model subgraphs in a medical KG and integrate it with a pre-trained language model to do knowledge generalization. |
| Outcome: | The proposed model achieves state-of-the-art on several medical NLP tasks . it improves on MedERNIE, and the proposed model is effective . |
AdaEdit: Advancing Continuous Knowledge Editing For Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge editing methods that can efficiently update knowledge in LLMs are limited due to budget constraints. |
| Approach: | They propose a method that can enhance the performance of edited LLMs in large-size continuous editing regimes. |
| Outcome: | Extensive empirical evaluations on multiple LLMs show that the proposed method outperforms existing methods without compromising the general abilities of these models. |
Unsupervised Improvement of Factual Knowledge in Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. |
| Approach: | They propose an approach that forces the model to prioritize informative words in a fully unsupervised way. |
| Outcome: | The proposed approach significantly improves the performance of pretrained language models on factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting. |
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LM) based on transformers generate plausible long texts . a discriminator-guided approach allows to apply constraints more finely and dynamically. |
| Approach: | They propose to use a discriminator-guided approach to generate constrained texts without fine-tuning the LM. |
| Outcome: | The proposed method is easier and cheaper to train than fine-tuning the LM. |
MBA-RAG: a Bandit Approach for Adaptive Retrieval-Augmented Generation through Question Complexity (2025.coling-main)
Copied to clipboard
| Challenge: | Existing RAG frameworks either indiscriminately perform retrieval or rely on rigid single-label classifiers to select retrieval methods. |
| Approach: | They propose a framework that dynamically selects the most suitable retrieval strategy based on query complexity. |
| Outcome: | The proposed framework achieves state-of-the-art results on multiple single-hop and multi-hop datasets while reducing retrieval costs. |
Progressive Class Semantic Matching for Semi-supervised Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent semi-supervised learning methods have achieved impressive performance . semi-controlled learning can be used to reduce the annotation cost of text classifiers . |
| Approach: | They propose a semi-supervised learning process that builds a standard K-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). |
| Outcome: | The proposed method improves baselines and overall is more stable. |
Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research shows that a multilingual pre-trained language model fine-tuned with one (source) language performs well on downstream tasks for non-source languages . However, there is a clear performance gap between the source and non-sourced languages - this gap can be reduced by reducing forgetting. |
| Approach: | They propose a method to fine-tune a multilingual pre-trained language model fine- tuned with one (source) language and four training policies to address the performance gap. |
| Outcome: | The proposed method outperforms baselines on the XNLI dataset by a clear margin. |
Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for neural machine translation (NMT) use encoder-only enhancement or rely on specific multilingual PLMs. |
| Approach: | They propose a monolingual PLM-sponsored NMT model that lets both encoder and decoder enjoy PLM enhancement. |
| Outcome: | The proposed model improves on IWSLT14 En-De, De-En, WMT14 En -De and En-Fr tasks and achieves new state-of-the-art. |
Unsupervised Abstractive Summarization of Bengali Text Documents (2021.eacl-main)
Copied to clipboard
Radia Rayan Chowdhury, Mir Tafseer Nayeem, Tahsin Tasnim Mim, Md. Saifur Rahman Chowdhury, Taufiqul Jannat
| Challenge: | Abstractive summarization systems are difficult to perform due to the unavailability of the parallel data for low-resource languages like Bengali. |
| Approach: | They propose a graph-based unsupervised abstractive summarization system in Bengali text documents that requires only a Part-Of-Speech (POS) tagger and a pre-trained language model trained on Bengali texts. |
| Outcome: | The proposed system outperforms baselines without human-annotated reference summaries on a human-random dataset with Bengali text. |
On the Computational Modelling of Michif Verbal Morphology (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing computational models of the verbal morphology of the Métis language are insufficient to model the language's unique phonological interactions. |
| Approach: | They propose a finite-state computational model of the verbal morphology of Michif . they use composed finite state transducers to model concatenative morphologies . |
| Outcome: | The proposed model is based on a series of finite-state transducers. |
Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to facilitate natural language understanding rarely involve commonsense or background knowledge. |
| Approach: | They propose a question-answering method that integrates multiple knowledge sources to boost performance. |
| Outcome: | The proposed method outperforms other competing methods on the CommonsenseQA dataset and achieves the new state-of-the-art. |
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. |
| Approach: | They propose a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena using a set of datasets and an evaluation procedure for diagnosing how well a language model captures reasoning skills. |
| Outcome: | The proposed model can diagnose model behavior and verify model learning quality. |
Dialogue-oriented Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PrLMs) have shown impressive improvements for various downstream tasks including various dialogue related ones. |
| Approach: | They propose to use pre-trained language models to simulate dialogue features on general plain text with common language model training objectives to improve performance. |
| Outcome: | The proposed method is fine-tuned on three public multi-turn dialogue datasets and achieves significant and consistent improvement over the plain PrLMs. |
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (N19-1)
Copied to clipboard
| Challenge: | Existing work on question-answering has limited training examples for RRC . question-announced questions are a key component of online commerce . |
| Approach: | They propose to turn customer reviews into a large source of knowledge that can be exploited to answer user questions. |
| Outcome: | The proposed approach improves review reading comprehension on popular language model BERT . it also improves aspect extraction and aspect sentiment classification tasks . |
F-Actor: Controllable Conversational Behavior in Full-Duplex Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Current spoken conversational systems lack customization capabilities, limiting their naturalness and usability. |
| Approach: | They propose an instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. |
| Outcome: | The proposed model requires just 2,000 hours of data to be trained under typical academic resource constraints. |
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, pre-trained language model (PLM) based metrics are widely adopted in text generation tasks. |
| Approach: | They propose to use PLMs to encode stereotypical societal biases in PLM-based metrics . they show that popular metrics exhibit higher social bias than traditional metrics based on 6 attributes . |
| Outcome: | The proposed method shows that PLM-based metrics exhibit higher social bias than traditional metrics on 6 attributes. |
Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained neural language models improve learning for various NLP tasks by fine-tuning them on task-specific training sets. |
| Approach: | They propose a meta-learning procedure to fine-tune neural language models on task-specific training sets. |
| Outcome: | The proposed procedure solves a group of similar NLP tasks on a text mining dataset. |
Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing graph neural networks (GNNs) teach message passing on a graph from text, resulting in a semantic gap between graph knowledge and text. |
| Approach: | They propose a framework to integrate external graph knowledge into chatbots by coagulating representations of both text and graph knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art (SOTA) baselines on dialogue generation. |
Language Models can be Deductive Solvers (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. |
| Approach: | They propose a novel language model that internalizes and emulates the reasoning processes of logical solvers and avoids parsing errors by learning strict adherence to solver syntax and grammar. |
| Outcome: | The proposed model outperforms state-of-the-art solver-augmented language models and few-shot prompting methods on public deductive reasoning benchmarks. |
Unified Demonstration Retriever for In-Context Learning (2023.acl-long)
Copied to clipboard
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, Xipeng Qiu
| Challenge: | In-context learning is a new learning paradigm where a language model conditions on a few input-output pairs (demonstrations) and a test input, and directly outputs the prediction. |
| Approach: | They propose a single model to retrieve demonstrations for a wide range of tasks by combining training signals from various tasks into a unified list-wise ranking formulation by language model’s feedback. |
| Outcome: | The proposed model outperforms baselines on 30+ tasks across 13 task families and multiple data domains. |
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind (2023.findings-acl)
Copied to clipboard
| Challenge: | Adaptation is a process in human communication by which a speaker tunes its language to that of a listener to achieve communicative success. |
| Approach: | They propose a visual-based referential game between a knowledgeable speaker and a listener with limited visual and linguistic experience to model this adaptation mechanism. |
| Outcome: | The proposed model improves on plug-and-play approaches to controlled language generation without finetuning the speaker’s underlying language model. |
Preserving Privacy Through Dememorization: An Unlearning Technique For Mitigating Memorization Risks In Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are trained on vast amounts of data, including sensitive information that poses a risk to personal privacy if exposed. |
| Approach: | They propose a novel unlearning approach that utilizes an efficient reinforcement learning feedback loop via proximal policy optimization to incentivize the LLMs to learn a paraphrasing policy to unlearn the pre-training data. |
| Outcome: | The proposed approach surpasses strong baselines and state-of-the-art methods in terms of its ability to generalize and strike a balance between privacy and LLM performance. |
Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain factuality of model knowledge across languages, and preserve parity of language model performance. |
| Approach: | They examine pretrained and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. |
| Outcome: | The proposed model shows different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and bottlenecks on a particular layer. |
Semantically-Prompted Language Models Improve Visual Descriptions (2024.findings-naacl)
Copied to clipboard
| Challenge: | Language-vision models have made significant progress in zeroshot vision tasks, but lack expressive visual descriptions. |
| Approach: | They propose a new method for generating visual descriptions with pre-trained language models and semantic knowledge bases. |
| Outcome: | The proposed method improves visual descriptions and achieves strong results on image-classification datasets. |
A Practical Examination of AI-Generated Text Detectors for Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to detect large language models are prone to misuse, such as generating fake news articles, facilitating academic plagiarism or spamming. |
| Approach: | They evaluate several popular detectors to evaluate their effectiveness against a range of domains, datasets, and models. |
| Outcome: | The proposed methods perform poorly in certain settings, with TPR@.01 as low as 0%. |
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)
Copied to clipboard
Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang
| Challenge: | Z-Code++ is a pre-trained language model optimized for abstractive text summarization. |
| Approach: | They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance. |
| Outcome: | The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings. |
Are Emergent Abilities in Large Language Models just In-Context Learning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models have been claimed to acquire certain capabilities without having been specifically trained on them. |
| Approach: | They propose a theory that explains emergent abilities by taking into account their potential confounding factors and rigorously substantiate this theory through over 1000 experiments. |
| Outcome: | The proposed theory proves that emergent abilities are not truly emergental, but result from a combination of in-context learning, model memory, and linguistic knowledge. |
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Semantic parsing aims to map a natural language sentence into a machine executable formal representation. |
| Approach: | They propose a structure-aware self-attention language model to capture structural information of target representations and propose incorporating it into a seq2seq model. |
| Outcome: | The proposed model improves the baseline model on four semantic parsing and Python code generation tasks. |
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? (2021.acl-long)
Copied to clipboard
| Challenge: | Analogies play a central role in human commonsense reasoning. |
| Approach: | They analyze the capabilities of transformer-based language models on an unsupervised task . they find off-the-shelf language models can identify analogies to a certain extent . |
| Outcome: | The proposed language models outperform word embedding models on an unsupervised task . the best results were obtained with GPT-2 and RoBERTa . |
Improving Segmentation for Technical Support Problems (2020.acl-main)
Copied to clipboard
| Challenge: | Technical support problems are long and complex and cannot be correctly parsed by tools designed for natural language. |
| Approach: | They propose a sequence labelling task and a supervised text segmentation approach to solve this problem. |
| Outcome: | The proposed approach improves on the downstream task of answer retrieval. |
Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent results show that the mix-of-experts architecture is parameter inefficient . large-scale pre-trained language models can achieve excellent performance in many NLP tasks. |
| Approach: | They propose to build a parameter-efficient mix-of-experts architecture by sharing information across experts. |
| Outcome: | The proposed architecture increases model capacity without increasing computation costs. |
LLM-Generated Passphrases That Are Secure and Easy to Remember (2025.findings-naacl)
Copied to clipboard
| Challenge: | Automated password generation is a cornerstone of IT security, but passphrases are often hard to remember and see limited adoption. |
| Approach: | They propose to use large language models to generate passphrases with sufficient entropy . they also use prompt construction to generate a grammatical model with sufficient output entropic . |
| Outcome: | The proposed methods show that the outputs are highly reliable and easy to remember . the proposed methods also show that prompt construction is important in steering the output topic and grammatical structure. |
Making Language Models Better Reasoners with Step-Aware Verifier (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models have made impressive progress in few-shot learning but still face difficulties in reasoning tasks such as GSM8K. |
| Approach: | They propose a new approach that uses a verifier to filter out incorrect answers based on a weighted voting scheme to improve reasoning ability of language models. |
| Outcome: | The proposed approach improves GSM8K reasoning rate by 17.9% to 58.1%. |
Korean Language Modeling via Syntactic Guide (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing research on pre-trained language models focuses on widely-used languages . however, not every language can benefit from such models due to computational resources . |
| Approach: | They propose to build a pre-trained language model that understands the linguistic phenomena in the target language with low resources. |
| Outcome: | The proposed model improves the performance of Korean language understanding tasks. |
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) capture a diverse range of linguistic and factual knowledge without the use of finetuning. |
| Approach: | They combine BERT with a traditional information retrieval step and a kNN search to improve the recall of facts encountered during training. |
| Outcome: | The proposed approach outperforms baselines on cloze-style QA by large margins without further training. |
A Data Cartography based MixUp for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Using MixUp, additional samples are generated during training by combining random pairs of training samples and their labels. |
| Approach: | They propose a new MixUp strategy that leverages Training Dynamics and allows more informative samples to be combined for generating new data samples. |
| Outcome: | The proposed method achieves competitive performance using a smaller subset of training data compared with strong baselines and yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings. |
ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | English and Chinese have seen the strong development of transformer-based language models for natural language processing tasks. |
| Approach: | They present a monolingual pre-trained language model for Vietnamese social media texts . they explore emotion recognition, hate speech detection, sentiment analysis, spam reviews detection . |
| Outcome: | The proposed model outperforms the existing models on Vietnamese social media tasks with fewer parameters. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
BECEL: Benchmark for Consistency Evaluation of Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing definitions of behavioural consistency are inconsistent across many studies. |
| Approach: | They propose a behavioural consistency model and propose behavioural taxonomy that classifies consistencies into several sub-categories. |
| Outcome: | The proposed model performs poorly on 19 test cases while exhibiting high inconsistency in many cases. |
Evaluating Biases in Context-Dependent Sexual and Reproductive Health Questions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | With the rise in accessibility of chat-based large language models, the public increasingly uses them as question-answering systems for personalized answers. |
| Approach: | They curate a dataset of sexual and reproductive healthcare questions dependent on age, sex, and location attributes and compare their outputs with and without demographic context to determine answer alignment . |
| Outcome: | The results show that young adult female users are favored in the model answers to underspecified questions in the healthcare domain. |
Compositional Demographic Word Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are usually derived from corpora containing text from many individuals . however, they cannot account for user-specific word preferences, such as using the same word in different ways across contexts. |
| Approach: | They propose a new form of personalized word embeddings that use demographic-specific word representations derived compositionally from full or partial demographic information for a user. |
| Outcome: | The proposed representations outperform generic representations on two English language tasks. |
Dual Adversarial Neural Transfer for Low-Resource Named Entity Recognition (P19-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is an important step in most natural language processing (NLP) applications. |
| Approach: | They propose a dual-adversarial neural transfer method for addressing low-resource Named Entity Recognition (NER) they propose 'Generalized Resource-Adversarial Discriminator' and 'accidental training' |
| Outcome: | The proposed method improves on low-resource Named Entity Recognition (NER) with two variants, i.e., DATNet-F and DATNET-P, and adversarial training is adopted to boost model generalization. |
A Pre-training Strategy for Zero-Resource Response Selection in Knowledge-Grounded Conversations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train retrieval-based dialogue systems rely on crowd-sourced data . however, it is difficult to collect large-scale dialogues that are grounded on background knowledge . |
| Approach: | They propose to decompose training of knowledge-grounded response selection into three tasks . they propose to combine query-passage matching task with query-dialogue history matching task . |
| Outcome: | Experimental results show that the proposed model can perform comparable to existing methods . the retrieval-based system can leverage background knowledge when conversing with humans . |
Unlearning Traces the Influential Training Data of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit remarkable abilities without being explicitly trained for such tasks. |
| Approach: | They propose a method that unlearns a test dataset and evaluates the unlearned model on training datasets. |
| Outcome: | The proposed method resembles UnTrac, while being efficient for massive training datasets. |
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs cannot generalize to domain-specific parsing tasks in a zero-shot setting. |
| Approach: | They propose a task-oriented parsing method that decomposes parse problem into abstractive and extractive question-answering problems. |
| Outcome: | The proposed method decomposes a parsing problem into abstractive and extractive question-answering (QA) problems. |
Compositionality of Complex Graphemes in the Undeciphered Proto-Elamite Script using Image and Text Embedding Models (2021.findings-acl)
Copied to clipboard
| Challenge: | a language model over sign images produces more interpretable results than a model over text . a new language model is developed to abstract from human annotators . |
| Approach: | They propose a language modeling architecture which operates over sequences of images or over multimodal sequences with associated labels. |
| Outcome: | The proposed language model can interpret signs in the undeciphered proto-Elamite script . it also provides a novel way to abstract away from biases introduced by human annotators. |
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for zero-shot text classification lack prompt engineering due to prompt brittleness . however, these methods are not effective for zero shot text classifications . |
| Approach: | They propose a method that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. |
| Outcome: | The proposed approach improves accuracy and reduces standard deviation by 98% . it maintains comparable performance even without a prompt, reducing the need for prompt engineering . |
Commonsense Knowledge Transfer for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have transformed the landscape of natural language processing. |
| Approach: | They propose a framework to transfer commonsense knowledge stored in a neural commonsensing model to a general-purpose pre-trained language model. |
| Outcome: | Empirical results show that the proposed framework improves the model’s performance on downstream tasks that require commonsense reasoning. |
Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing QA systems do not have commonsense knowledge or cannot reason with it. |
| Approach: | They propose to augment a general commonsense QA framework with a knowledgeable path generator by extrapolating existing paths from a KG with 'state-of-the-art' language model. |
| Outcome: | The generated paths are interpretable, novel, and relevant to the task. |
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (D19-1)
Copied to clipboard
| Challenge: | Variational Autoencoders are powerful language models and effective representation learning frameworks. |
| Approach: | They propose a fix for posterior collapse which improves held-out likelihood, reconstruction and latent representation learning . |
| Outcome: | The proposed fix significantly improves held-out likelihood, reconstruction, and latent representation learning compared with previous state-of-the-art methods. |
NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning, such as DeepSeek R1-Zero, highlight the effectiveness of incentive training, but these methods rely on external verifiers, which limits their applicability to domains like mathematics and coding, where such verifier is readily available. |
| Approach: | They propose a general reinforcement learning framework that requires only standard supervised fine-tuning data with no need for an external verifier. |
| Outcome: | The proposed framework outperforms the model of the same size distilled from large reasoning models such as DeepSeek R1 671B by 7.7%. |
Interpretable Math Word Problem Solution Generation via Step-by-step Planning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to solving math word problems focus on obtaining the correct answer. |
| Approach: | They propose a step-by-step planning approach for intermediate solution generation that strategically plans the generation of the next solution step based on the MWP and the previous solution steps. |
| Outcome: | The proposed approach improves the accuracy and interpretability of the solution on automatic metrics and human evaluation. |
PronouncUR: An Urdu Pronunciation Lexicon Generator (L18-1)
Copied to clipboard
| Challenge: | acoustic modeling, large text data and a pronunciation lexicon are the bottlenecks for speech recognition systems for resource scarce languages. |
| Approach: | They propose a grapheme-to-phoneme conversion tool that generates a pronunciation lexicon from a list of Urdu words. |
| Outcome: | The proposed tool predicts pronunciation of words using a LSTM-based model trained on a handcrafted expert lexicon of around 39,000 words and shows an accuracy of 64% upon internal evaluation. |
MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models fail to capture important semantic features of logic such as monotonicity and negation. |
| Approach: | They propose a modular step-by-step approach to natural language inference . they use a language model to generate edits to incrementally transform the premise into the hypothesis . |
| Outcome: | The proposed method outperforms baseline models in realistic cross-domain settings with improvements up to 12.6% (relative). |
Invariant Language Modeling (2022.emnlp-main)
Copied to clipboard
Maxime Peyrard, Sarvjeet Ghotra, Martin Josifoski, Vidhan Agarwal, Barun Patra, Dean Carignan, Emre Kiciman, Saurabh Tiwary, Robert West
| Challenge: | Existing methods to remove spurious correlations and biases involve expensive domain alignment. |
| Approach: | They propose a framework for learning invariant representations that generalize better across environments . they adapt a game-theoretic implementation of IRM to language models . |
| Outcome: | The proposed framework can remove structured noise, ignore correlations and achieve better generalization across environments. |
Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples (N19-1)
Copied to clipboard
| Challenge: | Neural abstractive summarization systems generate summary texts conditioned on the input source text, and have recently achieved high ROUGE scores on benchmark summarizing datasets. |
| Approach: | They propose to analyze existing neural abstractive summarization systems by comparing their performance to human-written summaries. |
| Outcome: | The proposed systems perform better than human-written summarizations on different datasets and show that they are able to understand deeper syntactic and semantic structures. |
StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing LM-based VC models require offline conversion from source semantics to acoustic features, limiting their deployment to real-time applications. |
| Approach: | They propose a streaming LM-based model for zero-shot voice conversion that uses a fully causal context-aware LM with a temporal-independent acoustic predictor to facilitate real-time conversion given arbitrary speaker prompts and source speech. |
| Outcome: | The proposed model achieves comparable performance to non-streaming VC systems while maintaining a fully causal context-aware LM with a temporal-independent acoustic predictor. |
Extremely Weakly-supervised Text Classification with Wordsets Mining and Sync-Denoising (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for weakly-supervised text classification use only class names as supervision . Existing approaches to classify texts without labeled data have significant flaws, including zero-shot instability and context-dependent ambiguities. |
| Approach: | They propose to use wordsets to generate pseudo-labels for unlabeled texts . they propose to train the classifier using a hybrid learning strategy called sync-denoising . |
| Outcome: | The proposed method outperforms all existing prompt and seed methods on 11 datasets by an impressive average of 8 points. |
Distilling ChatGPT for Explainable Automated Student Answer Assessment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing automated student answer assessment models lack explainable and faithful feedback. |
| Approach: | They propose a framework that leverages ChatGPT for student answer scoring and rationale generation. |
| Outcome: | The proposed method improves the overall QWK score by 11% compared to ChatGPT. |
SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing task-oriented dialog datasets do not situate the dialog in the user’s multimodal context. |
| Approach: | They propose to use a dataset to study multimodal task-oriented dialogs in the shopping domain to situate them in the user’s multimodal context. |
| Outcome: | The proposed dataset includes 11K task-oriented user->assistant dialogs (117K utterances) in the shopping domain, grounded in immersive and photo-realistic scenes. |
Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models Using Minimal Pairs (2024.lrec-main)
Copied to clipboard
| Challenge: | a new method is being developed to probe internal linguistic characteristics in neural language models layer by layer . |
| Approach: | They propose a method that uses minimal pairs benchmark to probe internal linguistic characteristics in neural language models layer by layer. |
| Outcome: | The proposed method captures grammaticality labels in language models layer by layer . it is based on the cognitive neurosciences of the brain and its representations as "neural activations". |
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text (2023.acl-long)
Copied to clipboard
| Challenge: | Self-supervised representation learning has proved to be a valuable component for out-of-distribution (OoD) detection with only the texts of in-difference (ID) examples. |
| Approach: | They propose a method that integrates strengths and weaknesses of both methods . they use a fine-tuned model as the teacher to teach a randomly initialized student model . |
| Outcome: | The proposed method outperforms human evaluators in the pair-expert task on the Human ChatGPT Comparison Corpus. |
Limitations of Autoregressive Models and Their Alternatives (2021.naacl-main)
Copied to clipboard
| Challenge: | Standard autoregressive language models only perform polynomial-time computation to compute probability of next symbol. |
| Approach: | authors propose alternative to standard autoregressive language models that use polynomial-time computation to compute probability of next symbol. |
| Outcome: | a large model size can grow superpolynomially in length, allowing it to store precomputed results and verify solutions. |
DEMix Layers: Disentangling Domains for Modular Language Modeling (2022.naacl-main)
Copied to clipboard
| Challenge: | Extensive experiments with autoregressive transformer LMs show that DEMix layers reduce test-time perplexity and increase training efficiency. |
| Approach: | They introduce a new domain expert mixture layer that enables conditioning a language model on the domain of the input text. |
| Outcome: | Experiments with 1.3B LMs show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation. |
SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task. |
| Approach: | They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines . |
| Outcome: | The proposed model improves sample-efficiency and performance over strong baselines. |
Finding Support Examples for In-Context Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning is a new learning paradigm where a language model observes a few examples and directly outputs the test input’s prediction. |
| Approach: | They propose a method to find “support examples” for in-context learning by filtering a training dataset and a progressive filtering process to filter out uninformative examples. |
| Outcome: | The proposed method outperforms baselines and shows that each component contributes critically to the improvements. |
Structural and Functional Decomposition for Personality Image Captioning in a Communication Game (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Personality image captioning (PIC) aims to describe an image with a natural language caption given a personality trait. |
| Approach: | They propose to use a communication game between a speaker and a listener to generate captions for PIC. |
| Outcome: | The proposed model achieves state-of-the-art performance for personal image captioning (PIC) the proposed model is based on a communication game between a speaker and a listener . |
HOTTER: Hierarchical Optimal Topic Transport with Explanatory Context Representations (2021.findings-emnlp)
Copied to clipboard
Sabine Wehnert, Christian Scheel, Simona Szakács-Behling, Maret Nieländer, Patrick Mielke, Ernesto William De Luca
| Challenge: | Natural language processing (NLP) is often the backbone of today’s systems for user interactions, information retrieval and others. |
| Approach: | They propose an extension to a specific emerging hybrid document distance metric which combines topic models and word embeddings. |
| Outcome: | The proposed method is competitive on public datasets and the language model BERT is used for a document categorization task. |
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)
Copied to clipboard
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi
| Challenge: | Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited. |
| Approach: | They propose an inference-time policy adapter which tailors a large base model without fine-tuning it. |
| Outcome: | The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4. |
EBERT: Efficient BERT Inference with Dynamic Structured Pruning (2021.findings-acl)
Copied to clipboard
| Challenge: | Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. |
| Approach: | They propose a dynamic structured pruning algorithm that prunes model weights at run-time . they propose to prune the unimportant heads in multi-head self-attention layers . |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods on different tasks. |
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve neural language models perform poorly on emerging data. |
| Approach: | They propose a lexical-level masking strategy to post-train a neural language model using static data from past years. |
| Outcome: | The proposed method outperforms existing methods on two pre-trained language models, two classification tasks, and four benchmark datasets. |
MUSCLE: A Model Update Strategy for Compatible LLM Evolution (2024.findings-emnlp)
Copied to clipboard
Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, Hadi Pouransari
| Challenge: | Large Language Models (LLMs) are regularly updated to enhance performance, typically through changes in data or architecture. |
| Approach: | They propose a method to minimize the extent of instance regression in model updates by training a compatibility adapter that can enhance task fine-tuned language models. |
| Outcome: | The proposed approach reduces regressions by up to 40% when updating LLMs to newer versions while maintaining overall performance gains. |
Data-Efficient Concept Extraction from Pre-trained Language Models for Commonsense Explanation Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract concepts from pre-trained language models are not suitable for commonsense explanation generation. |
| Approach: | They propose a method to extract the key explanation concept from pre-trained language models by fine-tuning it with 20% training data and using a metric to evaluate the retrieved concepts. |
| Outcome: | The proposed method improves evaluation metrics over pre-trained language models and the existing models. |
Disambiguating Homographs and Homophones Simultaneously: A Regrouping Method for Japanese (2024.lrec-main)
Copied to clipboard
| Challenge: | Using a method that re-groups surface forms into clusters representing synonyms, we examine how accurate such disambiguation can be. |
| Approach: | They propose to regroup homographs and homophones into clusters and use them to disambiguate them. |
| Outcome: | The proposed method is applied post-hoc to trained word embeddings in Japanese. |
Text Counterfactuals via Latent Optimization and Shapley-Guided Search (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using latent optimization and Shapley values, we generate a set of minimal modifications to the text to change the classifier's prediction. |
| Approach: | They propose to generate a counterfactual by making minimal modifications to the text to change the model's prediction. |
| Outcome: | The proposed approach achieves favorable performance compared to white-box and black-box baselines using human and automatic evaluations. |
MulDA: A Multilingual Data Augmentation Framework for Low-Resource Cross-Lingual NER (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual NER are labeled sequence translation and instance-based transfer via machine translation (MT) Existing methods to cross NER include label projection and labeling, but they are expensive and time-consuming. |
| Approach: | They propose a simple but effective labeled sequence translation method to translate source-language training data to target languages and avoids word order change and entity span determination. |
| Outcome: | The proposed method avoids word order change and entity span determination and can be generalized with the language-specific features from the target-language synthetic data and the language independent features from multilingual synthetic data. |
Improving Relation Extraction by Sequence-to-sequence-based Dependency Parsing Pre-training (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies show that dependency information is used only for encoder-only-based relation extraction tasks. |
| Approach: | They propose a syntax-aware seq2seq pre-trained model for relation extraction that incorporates dependency information into a seq2-trained language model by continual pre-training with a dependency parsing task. |
| Outcome: | The proposed model incorporates dependency information into a seq2seq pre-trained language model by continual pre-training with a generative sequence-to-sequence (sequ2sq)-based dependency parsing task. |
Activation Scaling for Steering and Interpreting Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a successful intervention should flip the correct with the wrong token, while remaining sparse. |
| Approach: | They propose to use activation scaling to flip the correct with the wrong token . they use gradient-based optimization to learn and evaluate a specific kind of efficient intervention . |
| Outcome: | The proposed method performs comparable with steering vectors but is much less minimal. |
Unleash GPT-2 Power for Event Detection (2021.acl-long)
Copied to clipboard
| Challenge: | Event Detection (ED) aims to recognize mentions of events and their types in text. |
| Approach: | They propose to exploit a pre-trained language model to generate training samples for ED. |
| Outcome: | The proposed model improves on multiple ED benchmark datasets and establishes state-of-the-art results. |
Toward A Digital Twin of U.S. Congress (2026.findings-acl)
Copied to clipboard
| Challenge: | a virtual model of congresspersons based on a collection of language models meets the definition of a digital twin. |
| Approach: | They propose to use a daily-updated dataset to generate tweets from congresspersons . they show that a modern language model equipped with subsets of this dataset produces Tweets that are indistinguishable from actual Tweets posted by their physical counterparts. |
| Outcome: | The proposed model produces Tweets that are indistinguishable from actual tweets posted by congresspersons. |
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train language models on diverse text corpora have brought up performance improvements on several natural language understanding (NLU) tasks. |
| Approach: | They propose a method to automatically generate domain- and task-adaptive maskings of a given text for self-supervised pre-training. |
| Outcome: | The proposed framework outperforms rule-based masking strategies on question answering and text classification datasets on which it outperformed rule-driven masking techniques. |
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
Data-Driven Pronunciation Modeling of Swiss German Dialectal Speech for Automatic Speech Recognition (L18-1)
Copied to clipboard
| Challenge: | a Swiss German speech recognizer is trained using a standard German annotation model. |
| Approach: | They propose to train a Swiss German speech recognition system using a standard German annotation model. |
| Outcome: | The proposed system is based on a standard German annotation model and a grapheme-to-phoneme conversion model. |
Reasoning Makes Good Annotators : An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract knowledge from unlabeled data generate noise labels. |
| Approach: | They propose an automatic task-specific rules distilling framework to generate a logic rule from unlabeled data. |
| Outcome: | The proposed framework could power the labeling ability by discovering reliable model-labeled data. |
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (2024.acl-long)
Copied to clipboard
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu
| Challenge: | Existing language models that use discrete representations for unified processing of various modalities are limited to text generation and do not include multimodal output. |
| Approach: | They propose a multimodal language model that utilizes discrete representations for unified processing of various modalities. |
| Outcome: | The proposed model can be trained stably without any alterations to existing models or training paradigms. |
Weak-to-Strong Honesty Alignment via Learning-to-Rank Supervision (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to enhance honesty with prompt engineering and fine-tuning are limited by annotated data. |
| Approach: | They propose a framework that enhances honesty through weak-to-strong generalization by training weak LLMs under weak supervision to improve their honesty. |
| Outcome: | The proposed framework improves honesty in large models even with limited label data. |
Logical Fallacy Detection (2022.findings-emnlp)
Copied to clipboard
Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, Bernhard Schoelkopf
| Challenge: | Existing language models perform poorly on logical fallacy detection . fallacious arguments can lead to disagreements, conflicts, endless debates, and a lack of consensus . |
| Approach: | They propose a task of logical fallacy detection and propose LogicClimate to detect fallacies in text. |
| Outcome: | The proposed task outperforms the best language model on Logic and LogicClimate . human reasoning is marred by logical fallacies, and some exacerbate misinformation . |
RelEdit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods struggle to reason about related conceptual knowledge effectively, despite a lack of model-level relational reasoning. |
| Approach: | They propose a benchmark to assess concept-level and instance-level relational reasoning abilities of edited models. |
| Outcome: | The proposed model obtains the best scores on the memory-based in-context editing baseline, MICE, suggesting a promising direction for model editing. |
HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn informative data representations on text-attributed hypergraphs struggle to capture full extent of hypergraph structural information and rich linguistic attributes inherent in the nodes attributes. |
| Approach: | They propose to augment a pre-trained BERT model with specialized hypergraph-aware layers for the task of node classification. |
| Outcome: | The proposed model outperforms existing methods on five challenging text-attributed hypergraph node classification benchmarks. |
Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution (2021.acl-long)
Copied to clipboard
| Challenge: | Abstractive summarization models have made great strides in recent years, but little is known about how they actually form summaries and how to understand where their decisions come from. |
| Approach: | They propose a two-step method to interpret summarization model decisions by categorizing each decoder decision into one of several generation modes. |
| Outcome: | The proposed method can identify phrases the summarization model has memorized and determine where in the training pipeline this memorization happened, and study complex generation phenomena on a per-instance basis. |
Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models for test case generation have improved branch coverage via prompt-engineered mutations, limiting their effectiveness for discovering subtle bugs and security vulnerabilities. |
| Approach: | They propose a program structure-aware LLM framework that integrates code property graphs and code semantics to condition test case generation on execution branches. |
| Outcome: | Experiments on real-world projects show that GLMTest improves branch accuracy from 27.4% to 50.2% on TestGenEval benchmark compared with state-of-the-art LLMs, i.e., Claude-Sonnet-4.5 and GPT-4o-mini. |
DischargeSim: A Simulation Benchmark for Educational Doctor–Patient Communication at Discharge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Discharge communication is a critical yet underexplored component of patient care, where the goal shifts from diagnosis to education. |
| Approach: | They propose a benchmark that evaluates large language models’ ability to act as personalized discharge educators. |
| Outcome: | Experiments with 18 LLMs show that model size does not always yield better education outcomes, highlighting trade-offs in strategy use and content prioritization. |
Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language models (LMs) and retrieval-augmented generation (RAG) have led to more capable chatbots and generative search engines. |
| Approach: | They propose to emulate the educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers by watching and steering the discourse among several LM agents. |
| Outcome: | The proposed system outperforms baseline methods on discourse trace and report quality and is preferred by 70% of participants over a search engine and 78% over sabota. |
From Alignment to Entailment: A Unified Textual Entailment Framework for Entity Alignment (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods encode the triples of entities as embeddings and learn to align the embeddables, which prevents the direct interaction between the original information of the cross-KG entities. |
| Approach: | They propose to transform the triples into unified textual sequences and model the EA task as a bi-directional textual entailment task between the sequences of cross-KG entities. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on five cross-lingual datasets and allows the mutual enhancement of the heterogeneous information. |
Defending against Insertion-based Textual Backdoor Attacks via Attribution (2023.findings-acl)
Copied to clipboard
| Challenge: | Textual backdoor attacks are vulnerable to backdoors and can be used to infect models trained on poisoned data. |
| Approach: | They propose an efficient attribution-based pipeline to defend against two insertion-based poisoning attacks, BadNL and InSent. |
| Outcome: | The proposed method can generalize sufficiently well in two common attack scenarios, which consistently improves previous methods. |
Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting modelgenerated texts from human texts are limited by the fact that absolute likelihood values of texts are bound to certain linguistic and cognitive constraints. |
| Approach: | They propose to use relative likelihood values instead of absolute ones to extract useful features from the spectrum-view of likelihood for the human-model text detection task. |
| Outcome: | The proposed method can reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies. |
Automatic Argument Quality Assessment - New Datasets and Methods (D19-1)
Copied to clipboard
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, Noam Slonim
| Challenge: | 6.3k arguments were collected from contributors of various levels, and are released as part of this work. |
| Approach: | They propose to use a language model to annotate arguments for argument ranking and argument-pair classification. |
| Outcome: | The proposed methods outperform state-of-the-art methods in the argument ranking task and argument-pair classification task. |
Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to unsupervised Chinese word segmentation require multiple inferences to perform word segmenting. |
| Approach: | They propose a method that integrates the segmentation signal from an unsupervised language model to a pre-trained BERT classifier under a pseudo-labeling framework. |
| Outcome: | The proposed method achieves state-of-the-art performance on the eight UCWS tasks while significantly reducing training time compared to previous approaches. |
Locally Differentially Private Document Generation Using Zero Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing privacy protections are limited by pretrained large language models . a new mechanism that leverages the power of large language model de-anonymization is proposed . |
| Approach: | They propose a locally differentially private mechanism that leverages pretrained large language models to counter author de-anonymization attacks by minimizing the impact on downstream utility. |
| Outcome: | The proposed mechanism reduces the success rate of de-anonymization attacks against IMDB datasets while minimizing the impact on downstream utility. |
CR-UTP: Certified Robustness against Universal Text Perturbations on Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing certified robustness methods for certifying input-specific text perturbations have shown promise in certifyling UTPs, but masking only adversarial words can eliminate the attack. |
| Approach: | They propose a method to certify a language model’s robustness against UTPs by using random smoothing. |
| Outcome: | The proposed method achieves high certified accuracy under extensive masking and achieves state-of-the-art results in multiple settings. |
Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs (D19-1)
Copied to clipboard
| Challenge: | a novel approach to paragraph planning involves a high-level control of different levels of relations between sentences . a proposed model with both forms of relations outperforms baselines in partially conditioned paragraph generation task . |
| Approach: | They propose two models that integrate human-created and latent relations into document-level language models . they focus on paragraph-level plan between sentences to produce coherent text . |
| Outcome: | The proposed models outperform baselines in partially conditioned paragraph generation task. |
Acquiring Frame Element Knowledge with Deep Metric Learning for Semantic Frame Induction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for semantic frame induction are labor intensive . a method that uses contextualized embeddings can be used to acquire frame element knowledge. |
| Approach: | They propose a method that applies deep metric learning to semantic frame induction tasks . they use a pre-trained language model to fine-tune frame-annotated models to perform argument clustering . |
| Outcome: | The proposed method achieves substantially better performance than existing methods on FrameNet. |
TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. |
| Approach: | They propose a language model for the low-resource language Turkish that is capable of both natural language understanding and generation tasks. |
| Outcome: | The proposed model outperforms multilingual models in understanding and generation tasks and competes with monolingual models for understanding tasks. |
Rethinking Masked Language Modeling for Chinese Spelling Correction (2023.acl-long)
Copied to clipboard
| Challenge: | Existing CSC models over-fit the error model while under-fitting the language model, resulting in poor generalization to out-of-distribution error patterns. |
| Approach: | They propose to use a multi-domain benchmark LEMON to assess the open domain generalization of Chinese Spelling Correction models. |
| Outcome: | The proposed method achieves state-of-the-art results on SIGHAN, ECSpell, and LEMON. |
Exploring the Value of Personalized Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | a subset of words belonging to specific psycholinguistic categories vary more in their representations across users . combining generic and personalized word embeddings yields the best performance . |
| Approach: | They propose personalized word embeddings and compare their performance to generic ones . they show that personalized word representations can be leveraged for improved performance . |
| Outcome: | The proposed model can be used for authorship attribution. |
Asymmetric Mutual Learning for Multi-source Unsupervised Sentiment Adaptation with Dynamic Feature Network (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work on pre-trained language models (PrLMs) on labeled sentiment datasets has shown significant improvements on widerange of NLP tasks, including sentiment classification. |
| Approach: | They propose a multi-source unsupervised sentiment adaptation problem with pre-trained features to exploit the extracted pre-train features for efficient domain adaptation. |
| Outcome: | The proposed model outperforms the state-of-the-art methods on multiple sentiment benchmarks and extensive ablation studies to verify the effectiveness of each module. |
Retrieving, Rethinking and Revising: The Chain-of-Verification Can Improve Retrieval Augmented Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent Retrieval Augmented Generation (RAG) aims to enhance Large Language Models . however, such approach can generate inconsistent answer with external references . |
| Approach: | They propose to integrate the verification module into the RAG to improve external retrieval correctness and internal generation consistency. |
| Outcome: | The proposed model can significantly surpass the state-of-the-art baselines using different LLM backbones. |
Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge (P19-1)
Copied to clipboard
| Challenge: | Existing methods to solve Winograd Schema Challenge use only knowledge embedded in text . this limits the performance of such models on the WSC problems. |
| Approach: | They propose to augment existing language models with a commonsense knowledge hunting module and an explicit reasoning module to extract the needed knowledge from text. |
| Outcome: | The proposed system improves on the language model based methods by 5.53% and 7.7% on the dataset. |
Transformer-based Live Update Generation for Soccer Matches from Microblog Posts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems to generate sports updates from tweets are not able to handle vast amounts of diverse tweets, and this remains a challenge for future studies. |
| Approach: | They propose to generate live updates for soccer matches from tweets using a large pre-trained language model and incorporate a classifier to control the number of updates and a mechanism to reduce redundancy of duplicate and similar updates. |
| Outcome: | The proposed system can generate live updates for soccer matches from tweets and achieve high performance by considering preceding updates. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
Improving Large-scale Paraphrase Acquisition and Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing Twitter-based paraphrase datasets lack quality definitions for identification and generation tasks. |
| Approach: | They propose to use two separate definitions of paraphrase for identification and generation tasks in existing Twitter-based paraphrase datasets. |
| Outcome: | The proposed model achieves state-of-the-art performance of 84.2 F1 for automatic paraphrase identification compared to other models fine-tuned on other corpora such as Quora, MSCOCO, and ParaNMT. |
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies focus on enhancing large-scale language models' reasoning abilities, but the research question of how to GSM8K Performance vs. computational cost remains. |
| Approach: | They propose to train small-scale language models with their own outputs to avoid relying on large models' outputs. |
| Outcome: | The proposed approach outperforms baseline models with comparable sizes while minimizing the required compute. |
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing paradigms for multi-task training involve a shared pre-trained language model and a small, thin network (head) given an input, a target head is the head that is selected for outputting the final prediction. |
| Approach: | They examine the behaviour of non-target heads when given input that belongs to a different task than the one they were trained for. |
| Outcome: | The non-target heads exhibit emergent behaviour, which may explain the target task, or generalize beyond their original task. |
FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for unlearning undesirable knowledge have overlooked complexity and interconnectedness of knowledge, authors say . previous studies have neglected the complex nature of knowledge and neglected its internal dependencies. |
| Approach: | They propose a new concept called superficial unlearning to evaluate faithfulness of unlearning in knowledge QA settings. |
| Outcome: | The proposed method shows significant effectiveness in real-world knowledge QA settings. |
When is a Language Process a Language Model? (2024.findings-acl)
Copied to clipboard
| Challenge: | In some pathological situations, such a stochastic process may "leak" probability mass onto the set of infinite strings. |
| Approach: | They propose to view a language model as a discrete stochastic process X t : t = = t + . |
| Outcome: | The proposed conditions of tightness are generalized to language models and the literature. |
Leveraging Web-Crawled Data for High-Quality Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, large language models are fine-tuned using expensive human-annotated data or GPT-4 generated data. |
| Approach: | They propose to use web-crawled data to train a language model on a smaller set of data . their results show that the model can convert web data with irregular formats into high-quality ones . |
| Outcome: | The proposed model outperforms open-source models larger than 32B and outperformed open-sourced models such as GPT-3.5. |
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain (2023.acl-long)
Copied to clipboard
| Challenge: | Increasing number of NLP benchmarks highlight need for multilingual models for job-related tasks. |
| Approach: | They introduce a language model called ESCOXLM-R that uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations taxonomy. |
| Outcome: | The proposed model outperforms XLM-R-large on short spans and entity-level and surface-level span-F1 tasks on entity- and surface level. |
Tree Transformer’s Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects (2024.acl-long)
Copied to clipboard
| Challenge: | Prepositional phrase attachment ambiguity is structural in nature, while garden path constructions are incremental in nature. |
| Approach: | They pretrain and evaluate an unsupervised Transformer model that induces tree representations internally and compare it to a pretrained supervised BiLSTM model. |
| Outcome: | The Tree Transformer model induces tree representations internally, but its parsing ability is inferior to the supervised BiLSTM model, and it is not as sensitive to lexical cues as other large LSTM models. |
Cross-sentence Pre-trained Model for Interactive QA matching (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for semantic matching do not examine each sentence individually, but consider syntactic context inside a sentence. |
| Approach: | They propose a semantic matching model that takes a cross-sentence context-aware architecture and incorporates a quantity of context information jump to facilitate attention weight formulation. |
| Outcome: | The proposed model outperforms state-of-the-art models on the Yahoo! community question dataset and the TREC library. |
Generative Multimodal Entity Linking (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing Entity Linking methods focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters. |
| Approach: | They propose a framework for multimodal entity linking based on Large Language Models (LLMs) that trains a feature mapper to enable cross-modal interactions. |
| Outcome: | The proposed framework achieves state-of-the-art on two well-established datasets with a performance gain of 7.7% on WikiDiverse and 8.8% on Wikileaks. |
Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020.acl-main)
Copied to clipboard
| Challenge: | a new method for combining multi-agent communication with traditional data-driven approaches to natural language learning is proposed . we combine the two types of learning with a goal of teaching agents to communicate with humans in natural language. |
| Approach: | They propose a method that combines traditional data-driven approaches to natural language learning with multi-agent self-play environments. |
| Outcome: | The proposed method outperforms other methods in communicating with humans in natural language. |
Data Similarity is Not Enough to Explain Language Model Performance (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models perform well on many but not all downstream tasks. |
| Approach: | They compare large language models with downstream benchmarks to determine whether similarity measures correlate with model performance. |
| Outcome: | The results show that similarity measures are not correlated with accuracy or each other in other benchmarks. |
Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot quantization methods are based on overfitting problem in adversarial learning process, leading to sub-optimal performance. |
| Approach: | They propose a zero-shot sharpness-aware quantization framework for the quantization of various PLMs by optimizing a minimax problem. |
| Outcome: | The proposed framework can achieve significant performance gains on discriminative and generative PLMs. |
Distilling Knowledge Learned in BERT for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT have revolutionized the state of the art in many language understanding tasks. |
| Approach: | They propose a conditional masked language modeling approach to fine tune BERT on target generation tasks by imposing global sequence-level supervision on conventional Seq2Seq models. |
| Outcome: | The proposed model outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. |
Text Augmented Open Knowledge Graph Completion via Pre-Trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to augment knowledge graph completion require factual triples or manual prompts to extract knowledge from a pre-trained language model. |
| Approach: | They propose a tool that generates quality query prompts and retrieves support information from large text corpora to probe knowledge from a pre-trained language model. |
| Outcome: | The proposed method outperforms embedding-based, graph-based and PLM-based methods on two benchmark datasets. |
HuatuoGPT, Towards Taming Language Model to Be a Doctor (2023.findings-emnlp)
Copied to clipboard
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li
| Challenge: | Experimental results show that the distilled language model outperforms its teacher model (ChatGPT) in most cases. |
| Approach: | They propose a Large Language Model (LLM) that leverages both distilled data from **ChatGPT** and real-world data from**doctors** in the supervised fine-tuning stage. |
| Outcome: | The proposed model outperforms the teacher model in most cases by using additional real-world data and RLMF to align the language model with the merits of both sources. |
Analysis of Plan-based Retrieval for Grounded Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large, parametric language models (LLMs) produce fluent text for many applications . hallucinations are generation of text that is factually correct and semantically plausible . |
| Approach: | They propose to use learning-tuned LLMs to infuse models with retrieval mechanisms to reduce hallucinations. |
| Outcome: | The proposed approach reduces the frequency of hallucinations by reducing the coverage of relevant facts and generating more informative responses while providing higher attribution rates. |
Disfluency Generation for More Robust Dialogue Systems (2023.findings-acl)
Copied to clipboard
| Challenge: | Disfluencies in user utterances can trigger a chain of errors impacting all the modules of a dialogue system. |
| Approach: | They propose to augment existing dialogue datasets with disfluent utterances by paraphrasing them into disfluente ones. |
| Outcome: | The proposed method improves dialogue state tracking and response generation by combining disfluent utterances with disfluency utteraces. |
UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on decoding word-level fMRI volumes from a restricted vocabulary. |
| Approach: | They propose an open-vocabulary task to bridge fMRI time series and human language . they use a pre-trained language model to construct a robust encoder for cognitive signals . |
| Outcome: | The proposed task bridges fMRI time series and human language with a baseline model. |
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023.acl-long)
Copied to clipboard
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
| Challenge: | Large “instruction-tuned” language models depend heavily on human-written instruction data . this limited quantity, diversity, and creativity hinders the generality of the tuned model . |
| Approach: | They propose a framework for improving instruction-following capabilities of pretrained language models by bootstrapping off their own generations. |
| Outcome: | The proposed framework outperforms existing public instruction datasets by 5% . it generates instructions, input, and output samples, then filters invalid or similar ones . |
KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Task-oriented dialogs (TOD) require a model to generate a response that optimizes for task-related metrics. |
| Approach: | They propose a faster generation procedure that samples from independent next-word distributions and introduce a fine-grained reward function to help the model focus on learning key information in a dialog. |
| Outcome: | The proposed algorithm achieves state-of-the-art performance on an offline task with 15% training time reduction compared to a standard RL algorithm using auto-regressive generation. |
Journalism-Guided Agentic In-context Learning for News Stance Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing stance detection research on news content is limited to short texts and high-resource languages. |
| Approach: | They propose a dataset for article-level stance detection that integrates viewpoints into recommendation algorithms and a framework that employs a language model agent to predict the stances of key structural segments. |
| Outcome: | The proposed framework outperforms existing methods in identifying article stances and uncovering patterns of media bias. |
Rethinking the Evaluation of In-Context Learning for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies evaluate In-context learning methods based on task performance . however, this evaluation protocol overlooks the significant cost associated with the demonstration configuration process . |
| Approach: | They propose a two-dimensional evaluation paradigm that considers both configuration costs and task performance. |
| Outcome: | The proposed evaluation paradigm can be applied to any ICL method as a plugin. |
Unified Representation for Non-compositional and Compositional Expressions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing representations of non-compositional language are based on BART, but they are not as accurate as the state-of-the-art IE representation model, GIEA. |
| Approach: | They propose a language model, PIER+, that builds on BART and can generate semantically meaningful and contextually appropriate representations for English potentially idiomatic expressions. |
| Outcome: | The proposed model achieves 33% higher homogeneity score on embedding clustering than BART, while sacrificing performance on NLU tasks (+/- 1% accuracy) |
RoBERTuito: a pre-trained language model for social media text in Spanish (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models have been used in many natural language processing tasks . some domain-specific models have shown to improve performance in some domains . however, for languages other than English, such models are not widely available . |
| Approach: | They present a pre-trained language model for user-generated text in Spanish . it is based on 500 million tweets and has some cross-lingual abilities . |
| Outcome: | The model outperforms models trained on over 500 million tweets on a benchmark in spanish and english. |
Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decoding strategies affect the probability distribution underlying the output of a language model and can therefore affect both generation quality and uncertainty. |
| Approach: | They investigate the impact of decoding strategies on uncertainty estimation in large language models . |
| Outcome: | The proposed methods improve the uncertainty estimation of large language models by reducing repetition. |
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions without human supervision. |
| Approach: | They propose to use a dataset of natural language instructions to generate large datasets with no human supervision. |
| Outcome: | The proposed dataset outperforms open-source models on various benchmarks, and is cost-effective. |
Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to improve language models require manual ranking and annotators. |
| Approach: | They propose a self-supervised text ranking approach for applying Proximal-Policy-Optimization to fine-tune language models while eliminating the need for human annotators. |
| Outcome: | The proposed method significantly outperforms baselines regarding BLEU, GLEU, and METEOR scores on three tasks and is consistent with humans. |
Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries (2025.emnlp-main)
Copied to clipboard
| Challenge: | To answer one-to-many factual queries, a language model must simultaneously recall knowledge and avoid repeating previous answers. |
| Approach: | They propose a promote-then-suppress mechanism that enables LMs to recall all answers and suppress previously generated ones. |
| Outcome: | The proposed model first recalls all answers, and then suppresses previously generated ones. |
Is It Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Text simplification and elaboration tasks are limited to only relatively altering the readability of texts to cater to a diverse audience. |
| Approach: | They propose to generate 8 versions of a text at different readability levels using ChatGPT and Llama-2 and introduce a two-step process to generate paraphrases. |
| Outcome: | The proposed task requires the generation of 8 versions at various target readability levels for each input text. |
TRN-R1-Zero: Text-rich Network Reasoning via LLMs with Reinforcement Learning Only (2026.acl-long)
Copied to clipboard
| Challenge: | Recent large language model-based approaches often overlook graph context or depend on distillation from larger models, limiting generalisation. |
| Approach: | They propose a framework for zero-shot reasoning on text-rich networks . they use a Neighbour-aware Group Relative Policy Optimisation objective . |
| Outcome: | The proposed framework optimises base LLMs using a Neighbour-aware group relative policy optimisation objective based on a novel margin gain metric for the informativeness of neighbouring signals . |
RaTEScore: A Metric for Radiology Report Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics to evaluate the quality of medical reports are limited due to the complexity of clinical free-form texts. |
| Approach: | They propose a new metric to assess the quality of medical reports generated by AI models. |
| Outcome: | The proposed metric is based on a medical NER dataset and trained on NER models . it aligns more closely with human preference than existing metrics, the authors show . |
CLAIR: Evaluating Image Captions with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing measures for image caption evaluation fail to capture dimensions of similarity . a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. |
| Approach: | They propose a method that leverages the zero-shot language modeling capabilities of large language models to evaluate captions. |
| Outcome: | The proposed method shows a stronger correlation with human judgments of caption quality compared to other measures. |
Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt compression reduces inference time and costs while maintaining informativeness for different usage scenarios. |
| Approach: | They propose a framework that adapts a smaller language model to compress prompts for a larger model on a new task without additional training. |
| Outcome: | The proposed framework outperforms two baseline models in four tasks . iteratively generates and selects effective compressed prompts as task-specific demonstrations . |
KPatch: Knowledge Patch to Pre-trained Language Model for Zero-Shot Stance Detection on Social Media (2024.lrec-main)
Copied to clipboard
Shuohao Lin, Wei Chen, Yunpeng Gao, Zhishu Jiang, Mengqi Liao, Zhiyu Zhang, Shuyuan Zhao, Huaiyu Wan
| Challenge: | Existing knowledge injection methods fail to understand the semantics of tweets . |
| Approach: | They propose a method to flexibly inject knowledge into a pre-trained language model and adaptively expand tweets context. |
| Outcome: | The proposed method is based on two training stages to flexibly inject knowledge into the pre-trained language model and adaptively expand tweets context. |
Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence (2023.findings-acl)
Copied to clipboard
| Challenge: | Sentence-level representations are beneficial for various natural language processing tasks. |
| Approach: | They propose a generative embedding inversion attack that reconstructs input sequences based only on their sentence embeddeds. |
| Outcome: | The proposed model outperforms previous embedding inversion attacks in classification metrics and generates coherent and contextually similar sentences as original inputs. |
Multi-Task Transfer Matters During Instruction-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction-tuning improves a model’s ability to learn in-context, but the mechanisms that drive in-constext learning are poorly understood. |
| Approach: | They propose to train a model on hundreds of tasks to improve its ability to learn in-context. |
| Outcome: | The proposed methods improve model transfer and in-context generalization, suggesting catastrophic forgetting may impact in-constext learning. |
Few-shot Reranking for Multi-hop QA via Language Model Prompting (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multi-hop QA with open-domain questions require a large number of labeled question-document pairs for retrieval. |
| Approach: | They propose a language-based prompt for multi-hop path reranking that relies on language model prompting to generate a relevance score between a question and the path. |
| Outcome: | The proposed method yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples. |
Synthesize, if you do not have: Effective Synthetic Dataset Creation Strategies for Self-Supervised Opinion Summarization in E-commerce (2023.findings-emnlp)
Copied to clipboard
Tejpalsingh Siledar, Suman Banerjee, Amey Patil, Sudhanshu Singh, Muthusamy Chelliah, Nikesh Garera, Pushpak Bhattacharyya
| Challenge: | Existing approaches to generate general and aspect-specific opinion summarization are limited due to their reliance on human-specified aspects and seed words. |
| Approach: | They propose synthetic dataset creation approaches for general and aspect-specific opinion summarization . general opinion summaries struggle to generate faithful to the input reviews, they say . aspect- specific opinion summarisation models are limited due to reliance on human-specified aspects . |
| Outcome: | The proposed approach outperforms existing models on three e-commerce test sets on general and aspect-specific opinion summarization. |
Recursive Question Understanding for Complex Question Answering over Heterogeneous Personal Data (2025.findings-acl)
Copied to clipboard
| Challenge: | a novel method for question answering over mixed sources, like text and tables, has been developed for question-answering . personal information is a prominent case of such heterogeneous data, such as calendar entries, workout statistics, shopping records, streaming history, and more. |
| Approach: | They propose a method that creates an executable operator tree for a given question . they use recursive decomposition to decompose a question into an operator tree . |
| Outcome: | The proposed method outperforms methods based on verbalization or translation . it can be executed on user devices and yields a traceable answer . |
Defending LLMs against Jailbreaking Attacks via Backtranslation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advancement in large language models (LLMs) has shown their extensive applications and transformative potential to reshape people's lives. |
| Approach: | They propose a method which uses backtranslation to infer an input prompt from an input input prompt and then run it again on the backtranslated prompt. |
| Outcome: | The proposed method outperforms baselines and has little impact on the generation quality for benign input prompts. |
MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing retrieval-augmented models typically retrieve information from a single type of knowledge source. |
| Approach: | They propose an efficient memory-augmented transformer to retrieve relevant knowledge from multiple knowledge sources. |
| Outcome: | The proposed model outperforms existing retrieval-augmented models on popular QA benchmarks in terms of accuracy and speed. |
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on the ethical and ecological impact of pre-trained language models raise questions about the temporal, financial, and environmental aspects of such models. |
| Approach: | They propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLMs. |
| Outcome: | The proposed model is compared with classical multilingual models and is ethically virtuous. |
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent smaller language models rely on synthetic data generated using larger Language models. |
| Approach: | They propose a method for generating synthetic data that enhances diversity through meta-prompting . they use 25 million tokens of synthetic data generated by a language model orchestrated by multiple “expert” LLM agents to collaboratively generate data. |
| Outcome: | The proposed method outperforms the base LLM in Finance and Biomedicine with 25 million tokens of synthetic data. |
SocialGaze: Improving the Integration of Human Social Norms in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasingly, large language models (LLMs) are able to understand and rationalize socially acceptable behaviors, but they are often misaligned with human consensus. |
| Approach: | They propose a multi-step prompting framework that verbalizes a social situation from multiple perspectives before forming a judgment. |
| Outcome: | The proposed framework improves the alignment with human judgments by up to 11 F1 points with the GPT-3.5 model. |
MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for retraining from scratch are limited and only work on the recall of edited facts. |
| Approach: | They propose a benchmark method that allows users to ask multi-hop questions to assess whether edited models correctly answer questions where the answer should change as an entailed consequence of edited facts. |
| Outcome: | The proposed method outperforms existing models and scales well with LLMs (up to 175B) it is based on a memory-based approach that stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts. |
Breaking the "Provable Security": Detecting Finite-Precision Artifacts in LLM-based Steganography via Low-Probability Vanishing (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have fostered a new class of generative linguistic steganography, claim “provably secure” by theoretically aligning the stego distribution with the language model’s natural distribution. |
| Approach: | They propose a framework that transforms the detection task from semantic classification to a statistical audit of the sampling mechanism. |
| Outcome: | The proposed framework breaks the security of AC and Meteor with high detection accuracy, whereas state-of-the-art semantic steganalyzers degrade to random guessing. |
Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can explain grammatical rules, but fail to apply those rules when judging sentence acceptability. |
| Approach: | They propose a grammar prompting paradigm that feeds an LLM's metalinguistic explanation back to the target model before deciding which sentence of a minimal pair is grammatical. |
| Outcome: | The proposed model improves on the English BLiMP, Chinese SLING, and Russian RuBLimp benchmarks. |
Learning Subjective Label Distributions via Sociocultural Descriptors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches aggregate annotator judgements (labels), ignoring minority perspectives, and overlooking the influence of sociocultural context behind such annotations. |
| Approach: | They propose a framework where subjectivity in binary labels is modeled as an empirical distribution accounting for the variation in annotators through human values extracted from sociocultural descriptors using a language model. |
| Outcome: | The proposed model yields well-calibrated toxicity distribution predictions across binary toxicity labels, which are further used for majority label prediction across cultural subgroups. |
Composable Text Controls in Latent Space with ODEs (2023.emnlp-main)
Copied to clipboard
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, Zhiting Hu
| Challenge: | Existing approaches to composable text operations often require plug-and-play . a single LM can perform arbitrary text operation composition in the latent space . |
| Approach: | They propose an efficient approach for composable text operations in the latent space of text . they connect pretrained LMs to the laten space and adapt them to the space . |
| Outcome: | The proposed approach improves on existing methods in the latent space of text. |
The Linearity of the Effect of Surprisal on Reading Times across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large amount of insight into human language processing can be gleaned by studying word-by-word processing difficulty. |
| Approach: | They extend the study by examining eyetracking corpora of seven languages . they find evidence for superlinearity in some languages, but highly sensitive to language models . |
| Outcome: | The study extends existing studies on english to Danish, Dutch, English, German, Japanese, Mandarin, and Russian. |
Attack as Defense: Safeguarding Large Vision-Language Models from Jailbreaking by Adversarial Attacks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | adversarial vulnerabilities in vision-language systems pose a challenge to reliability of large systems . typographic manipulations and adversarial perturbations can bypass language model defenses . |
| Approach: | They propose a method that embeds perturbations in vision to disrupt attacks . they use cross-modal interactions to enhance adversarial robustness through perturbations . |
| Outcome: | The proposed approach reduces attack success rates for typographic attacks and adversarial perturbations by integrating visual defenses into the model. |
On the Relation Between Fine-Tuning, Topological Properties, and Task Performance in Sense-Enhanced Embeddings (2025.acl-long)
Copied to clipboard
| Challenge: | Enhanced word embeddings do not align well with word senses, resulting in poor performance on word sense identification tasks. |
| Approach: | They propose to use two methods to fine-tune embeddings to identify the topological properties that contribute to sense-enhanced embeddables. |
| Outcome: | The proposed methods improve the embeddings’ ability to capture nuanced semantic distinctions while reducing their expressiveness. |
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law (2025.acl-long)
Copied to clipboard
Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen
| Challenge: | Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. |
| Approach: | They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability. |
| Outcome: | The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities. |
Inductive Linguistic Reasoning with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. |
| Approach: | They propose to generate analogical exemplars with a language model and apply them in-context with target language exemplar. |
| Outcome: | The proposed method can be applied to other tasks present in Linguistics Olympiad competitions and achieves state-of-the-art results across nearly all problem types and difficulty levels in the LINGOLY dataset. |
Reranking-based Generation for Unbiased Perspective Summarization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on traditional metrics for measuring key attributes such as coverage and faithfulness without verifying their applicability. |
| Approach: | They propose to use human annotations to measure perspective summary quality and reranking-based methods yield strong results. |
| Outcome: | The proposed methods show that they perform well with synthetically generated and reranking-labeled data. |
Syntactic Control of Language Models by Posterior Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Syntactic control of generated text is crucial for many domain-specific applications of language models, where structural constraints, such as formality, grammatical correctness, or adherence to a given template, can significantly affect usability and readability. |
| Approach: | They propose to use posterior inference to enforce target constituency structure by sampling from a proposal distribution with a syntactic tagger to ensure that each generated token aligns with the desired syntakic structure. |
| Outcome: | The proposed method increases the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model’s fluency. |
MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context. |
| Approach: | They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans. |
| Outcome: | The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases. |
The Ethical Question – Use of Indigenous Corpora for Large Language Models (2024.lrec-main)
Copied to clipboard
Linda Wiechetek, Flammie A. Pirinen, Børre Gaup, Trond Trosterud, Maja Lisa Kappfjell, Sjur Moshagen
| Challenge: | Creating language technology based on language data is becoming more popular . indigenous language resources are not comparable in that they would encode the most recent normativised language . |
| Approach: | They describe an ethical way to work with indigenous languages based on language data . they say data driven methods make assumptions based upon majority languages they work with . authors say data-driven methods are not ethical or beneficial . |
| Outcome: | The proposed method is ethical and sustainable, and can be applied to indigenous languages in an ethical way. |
Accelerated Test-Time Scaling with Model-Free Speculative Sampling (2025.emnlp-main)
Copied to clipboard
Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin, Aram Galstyan, Sravan Babu Bodapati
| Challenge: | Language models have demonstrated remarkable capabilities in reasoning tasks through test-time scaling techniques like best-of-N sampling and tree search. |
| Approach: | They propose a model-free speculative decoding approach that exploits redundancy in reasoning trajectories to achieve significant acceleration without compromising accuracy. |
| Outcome: | The proposed approach reduces inference latency by 60-65% while maintaining accuracy. |
Self-Guided Plan Extraction for Instruction-Following Tasks with Goal-Conditional Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | a framework for instruction-following tasks is proposed for instruction following tasks . previous methods rely on expert trajectories and learn directly from the agent's own interactions with the environment without expert supervision. |
| Approach: | They propose a framework for instruction-following tasks that enables a language model to generate and refine high-level plans through a self-learning mechanism. |
| Outcome: | The proposed framework adheres to instructions more strictly than baseline methods while showing strong generalization to previously unseen instructions. |
A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | a novel multi-task learning framework for domain-specific natural language understanding tasks addresses these limitations by combing multiple tasks into a single framework. |
| Approach: | They propose a multi-task learning framework that decomposes the language model into modular skill components and employs a dynamic, learnable skill-combination mechanism to adaptively handle diverse tasks. |
| Outcome: | The proposed framework surpasses conventional multi-task learning approaches in performance. |
Text Embedding as Treatment: A Meta Causal Approach for Robust Sentiment Classification (2026.findings-acl)
Copied to clipboard
Fengxiang Cheng, Chuan Zhou, Xiang Li, Haoxuan Li, Wen-li Wang, Jinkun Chen, Mingming Gong, Kun Zhang
| Challenge: | Existing methods for sentiment classification use binary treatment of words . Existing approaches limit generalizability to novel words and low-frequency words if there is a word in a sentence that is not treated . |
| Approach: | They propose a meta-causal approach that uses a single training task to identify causal words for arbitrary words. |
| Outcome: | The proposed method reduces the spurious correlation between word treatment and sentiment classification by removing words with low treatment effects from a pre-trained language model. |